Project Abstract

Existing 3D shape datasets in the research community are generally limited to objects or scenes at the home level. City-level shape datasets are rare due to the difficulty in data collection and processing. However, such datasets uniquely present a new type of 3D data with a high variance in geometric complexity and spatial layout styles, such as residential/historical/commercial buildings and skyscrapers. This work focuses on collecting such data, and proposes city generation as new tasks for data-driven content generation. In addition, a proposing new city-level generation models is also included in this project.

Project Description & Overview

As an important arena for human activities, cities have been a focal point of research. Alongside the rapid advancement of image/video generation, data-driven 3D city generation has become more feasible and appealing because of 1) the increasing availability of city-level remote sensing, and 2) the intensification of data-driven methods in architecture and urban planning.

While deep generative models are successful for various data modalities, including language, audio, image, video, and even point clouds, the limited publicly available 3D real-world city datasets makes it difficult to apply deep generation methods towards city-level geometric generation.

The team will work through 3 stages:

• General city-level generation model as baseline (60%)
  • The team should try to retrain our baseline models on Realcity3D datasets and extend our AETree baseline to more blocks.
  • The team should propose some city-level generation models (should include 2D and 3D data) and train on Realcity3D datasets.
• Literature review (10%)
  • The team will complete a review on data-driven generation methods and datasets for geometric generation.
• Data collection and cleaning on other cities (30%)
  • The team should follow the same way of collecting data as the author did to collect more data on other cities in order to increase the diversity of the Realcity3D dataset.

Datasets

Existing Realcity3D data of NYC and Zurich will be used for stage 1. We need to collect more data of other cities in stage 3.

Competencies

• Machine learning
  • Supervised learning
  • Self-supervised learning
  • Computer vision experience
  • 2D and 3D data processing
  • Good code and data management skills
  • Python basic and PyTorch DL library
• Technical experience
  • Python programming (required for >=2 team members)
  • Data processing pipelines
  • Documentation
• Data management experience
• Privacy and data
• Data collection and analysis
• Ethics

Learning Outcomes & Deliverables

The team will be using a broad range of deep learning models that will result in proven abilities in: computer vision, content generation, urban planning, and machine learning.

The expected deliverables for each project stage are:

• Retrain and extend our AEtree baselines and get reasonable results.
• Propose some generation models for city-level geometric generation and get some baseline results.
• Extend Realcity3D data to more city data.

All deliverables will be based around Jupyter notebooks and committed to a well documented public GitHub repository.

Students

Qingrui Duan, Xianyou Li, Zirui Liu, Jingyuan Yu