Executive summary
This article shows how teams can design a data lake that spans multiple Alibaba Cloud regions while keeping data in its native object format. By placing files in Object Storage Service (OSS), describing them in Data Lake Formation (DLF), and using MaxCompute external schemas, organizations can run SQL queries without loading data into a traditional warehouse. The guide outlines the technical foundations, architecture, step‑by‑step configuration, security model, typical enterprise scenarios, and operational tips, all aimed at delivering a scalable, cost‑effective analytics platform that can grow with data volume and query concurrency.
Introduction
Modern analytics workloads demand storage that is cheap, durable, and globally accessible. Alibaba Cloud provides OSS as a core repository for binary data such as Parquet, CSV, and JSON files. To make that data queryable, DLF offers a catalog that stores table definitions, partitions, and schema information. MaxCompute—Alibaba’s serverless big‑data processing engine—can read OSS objects directly when an external schema maps a DLF catalog to a MaxCompute project. This pattern, often described as a lakehouse, eliminates the need to ingest data into a separate warehouse and enables low‑cost, high‑performance analytics while preserving the original file format.
Technical fundamentals
- OSS stores raw byte files. It is region‑specific; objects in one region cannot be accessed by services in another region without cross‑region replication.
- DLF maintains metadata (catalog, database, table definitions). The lakehouse feature that links DLF and OSS is documented as lakehouse 2.0 and is only available in a subset of regions: China (Hangzhou, Shanghai, Beijing, Zhangjiakou, Shenzhen, Hong Kong), Singapore, and Germany (Frankfurt).
- MaxCompute executes SQL queries. When an external schema is created, MaxCompute reads DLF metadata at query time and fetches the matching OSS objects. Permissions are enforced by a RAM role that can read both DLF and OSS, and by MaxCompute‑level permissions on the mapped objects.
- External schema vs. managed lake table: An external schema provides schemaless query → external table → managed lake table progression of data‑management strength. Schemaless query reads files without a predefined schema, external schema maps DLF metadata, and managed lake tables store data and metadata fully inside MaxCompute, offering ACID guarantees.
Architecture and components
The lakehouse architecture consists of four logical components:
- OSS bucket – holds data files (Parquet, CSV, JSON).
- DLF catalog/database – describes the files, their partitions, and schema.
- MaxCompute project – runs analytics jobs.
- External schema – a bridge that binds a DLF catalog/database to the MaxCompute project.
Below is a simple diagram of the data flow.
+-------------------+ +-------------------+ +-------------------+
| OSS Bucket | <---> | DLF Catalog | <---> | MaxCompute Project |
| (raw Parquet/CSV) | | (metadata) | | (SQL engine) |
+-------------------+ +-------------------+ +-------------------+
| ^
| |
+----------- SQL Query ---------+
Detailed implementation approach
- Create OSS bucket in a supported region and upload data files using the console, SDK, or command‑line tools.
- Create DLF catalog and database that point to the OSS location. The catalog stores table definitions that reference the OSS path and optional partition keys.
- Provision a MaxCompute project in the same region. Choose a project name that reflects the business domain for easier governance.
- Authorize access:
- Ensure the MaxCompute project owner also owns the OSS bucket and DLF catalog, or use the console’s one‑click authorization which creates the necessary RAM role automatically.
- Attach a RAM role that grants
Read on both DLF and OSS. The role should be scoped to the specific catalog and bucket to follow the principle of least privilege.
- Create external schema using the documented SQL syntax:
SET odps.namespace.schema=true;
CREATE EXTERNAL SCHEMA IF NOT EXISTS my_lake_schema
WITH oss ON 'mycatalog.dlf_database';
The command maps the DLF catalog mycatalog and database dlf_database to the external schema my_lake_schema inside the MaxCompute project.
- Query data without copying:
USE my_lake_schema;
SELECT * FROM sales_parquet WHERE region = 'APAC' LIMIT 100;
MaxCompute reads the DLF metadata at execution time, resolves the OSS objects, and streams the rows directly to the query engine. No intermediate copy is performed, which preserves storage efficiency and reduces latency.
Architecture and data‑flow
The data‑flow can be described in three stages:
- Ingestion – Files are written to OSS by applications, ETL jobs, or streaming services. Data may arrive in batches or continuously, but each file is immutable after upload.
- Catalog update – DLF is refreshed either manually or through automated crawlers that register new partitions and schema changes. The crawler can be scheduled to run hourly, ensuring that newly arrived files become instantly queryable.
- Query execution – MaxCompute resolves the external schema, pulls the latest DLF metadata, and reads the OSS objects directly. Because the data stays in OSS, compute resources are only consumed during query time, keeping overall cost low.
Security and governance
- RAM role: Must have
dlf:Read and oss:Read actions. The role is assumed by the MaxCompute project at query time, providing a clear audit trail.
- Authorization mapping: When the same Alibaba Cloud account owns OSS, DLF, and MaxCompute, the console provides a one‑click mapping that automatically creates the RAM role and attaches it to the project. In multi‑account environments, administrators must manually grant the role to the target account.
- Data governance: DLF acts as the single source of truth for schema and partition information. Any drift between DLF metadata and OSS objects can cause query failures, so regular synchronization checks are recommended. Exporting DLF definitions to a version‑controlled repository helps with change management.
- Network isolation: OSS can be configured with VPC endpoints to keep traffic within a private network, and MaxCompute can be restricted to specific VPCs via security groups. This prevents data from traversing the public internet and satisfies compliance requirements for many enterprises.
Enterprise use cases
| Use case | How the lakehouse helps |
|---|
| Click‑stream analytics | Raw JSON logs land in OSS; DLF catalogs them; analysts run ad‑hoc SQL on MaxCompute without ETL. |
| Financial reporting | CSV statements are stored in OSS; DLF defines column types; MaxCompute produces regulatory reports on demand. |
| Machine‑learning feature store | Parquet feature files reside in OSS; DLF maintains versioned schemas; training jobs query directly from MaxCompute. |
Benefits and limitations
Benefits
- No data duplication; queries run directly on object storage, preserving the original format.
- Separation of storage (OSS) and compute (MaxCompute) allows independent scaling of each layer based on workload characteristics.
- Fine‑grained access control via RAM roles and MaxCompute permissions ensures that only authorized principals can view or query specific datasets.
- Compatibility with existing data formats (Parquet, CSV, JSON) means migration from on‑premise data lakes is straightforward.
Limitations
- The lakehouse feature (DLF + OSS external schema) is only available in the listed regions; deployments outside those regions must use alternative solutions such as manual catalog synchronization.
- Metadata must stay in sync; DLF does not automatically enforce consistency, so operational processes are required to detect and resolve drift.
- Managed lake tables provide stronger data management (e.g., ACID guarantees) but require data to be copied into MaxCompute, incurring additional storage cost.
- The older Data Lakehouse Solution 1.0 external project is being phased out; existing users need to migrate to external schemas to stay on a supported path.
Operational considerations
- Version compatibility: The MaxCompute client must be version 0.46.8 or later to support
SHOW TABLES IN an external schema. Older clients will report syntax errors.
- Monitoring: Enable OSS request logs and DLF audit trails to track data access patterns and catalog changes. Integrate these logs with CloudMonitor for alerting on unusual activity.
- Cost management: OSS storage is billed per GB‑month; MaxCompute charges are based on compute resources used during query execution. Because data is not duplicated, storage cost remains low, and compute cost scales only with query concurrency.
- Backup strategy: OSS provides versioning and cross‑region replication; DLF metadata should be exported regularly (for example, nightly) to an Object Storage bucket in a different region to protect against regional failures.
- Performance tuning: Partition data on high‑cardinality columns (such as date or region) to enable predicate push‑down. MaxCompute can prune irrelevant OSS objects early, reducing I/O.
Best practices
- Co‑locate resources – Keep OSS bucket and DLF catalog in the same region to avoid cross‑region latency and additional data transfer charges.
- Dedicated RAM roles – Use a distinct RAM role for each MaxCompute project or business domain. This limits the blast radius if credentials are compromised.
- Automate catalog updates – Deploy a scheduled job (for example, an Function Compute task) that scans OSS directories for new partitions and issues DLF
ADD PARTITION commands.
- Choose the right table type – Prefer managed lake tables for tables that require frequent updates, transactional guarantees, or row‑level security. Use external schemas for immutable or append‑only data where low‑cost storage is paramount.
- Validate schema alignment – After bulk uploads, run a small
SELECT COUNT(*) or DESCRIBE query to verify that the DLF schema matches the actual file structure before releasing the data to downstream users.
- Audit regularly – Review RAM role permissions quarterly and reconcile them with the principle of least privilege. Remove any stale roles that no longer map to active projects.
Analysis
The source material confirms that the lakehouse architecture relies on three tightly coupled services: OSS for storage, DLF for metadata, and MaxCompute for compute. The external schema mechanism is the only documented way to query OSS data without copying it into MaxCompute. Regions outside the documented list lack this capability, which is a critical planning factor for global teams. Security is handled through RAM roles, and the one‑click console mapping simplifies initial setup but does not replace the need for ongoing permission audits, network isolation checks, and metadata consistency verification.
What the sources do not settle
- Performance benchmarks: The documentation does not provide quantitative latency or throughput numbers for queries against OSS via external schemas, so teams must conduct their own testing under realistic workloads.
- Metadata drift detection: While DLF must stay aligned with OSS objects, the notes do not specify built‑in mechanisms for detecting mismatches. Implementing periodic checksum validation or using OSS event notifications to trigger DLF updates can fill this gap.
- Cross‑region federation: There is no guidance on how to query OSS data that resides in a different region from the MaxCompute project, implying that such scenarios are unsupported and would require data replication or a separate lakehouse deployment per region.
Conclusion
Designing a scalable data lake on Alibaba Cloud involves storing raw files in OSS, cataloging them with DLF, and querying them directly from MaxCompute through an external schema. This approach offers low storage cost, flexible compute scaling, and fine‑grained security, but it is limited to the regions where the lakehouse feature is released. Proper governance, synchronization, and role management are essential to maintain a reliable analytics platform that can serve diverse enterprise workloads.
References
- Data Lake Analytics (Data Lakehouse) | Alibaba Cloud
- DLF and OSS external schema | Alibaba Cloud
- Query CSV data in OSS with DLF | Alibaba Cloud
These published CloudExchangers articles are further desk reading. They are not sources for the technical claims above.