前言

学习总结Hudi Spark SQL Call Procedures，Call Procedures在官网被称作存储过程（Stored Procedures），它是在Hudi 0.11.0版本由腾讯的ForwardXu大佬贡献的，它除了官网提到的几个Procedures外，还支持其他许多Procedures命令。本文先学习其中的几个我觉得比较常用的命令，主要是查询统计表路径下的各种文件信息。

版本

Hudi master 0.13.0-SNAPSHOT
Spark 3.1.2 (实际上所有Hudi支持的Spark版本都支持Call Procedures)
Kyuubi 1.5.2 (使用Kyuubi是因为返回结果可以展示列名，Spark自带的spark-sql不返回列名)

参数形式

按名称传递参数，没有顺序，可以省略可选参数

CALL system.procedure_name(arg_name_2 => arg_2, arg_name_1 => arg_1, ... arg_name_n => arg_n)

按位置参数传递参数，有顺序，可以省略可选参数

CALL system.procedure_name(arg_1, arg_2, ... arg_n)

支持的Procedures命令

我们可以在类HoodieProcedures获取对应版本支持的所有的Procedures命令。目前支持如下：

show_fs_path_detail
show_bootstrap_partitions
repair_deduplicate
create_metadata_table
stats_file_sizes
validate_metadata_table_files
show_commit_partitions
show_commit_extra_metadata
show_table_properties
run_clustering
run_bootstrap
show_commit_files
run_clean
show_rollback_detail
rollback_to_savepoint
show_fsview_all
show_compaction
copy_to_temp_view
show_invalid_parquet
delete_savepoint
show_bootstrap_mapping
show_archived_commits
show_fsview_latest
show_metadata_table_files
export_instants
show_commits_metadata
rollback_to_instant
delete_metadata_table
delete_marker
show_metadata_table_stats
sync_validate
copy_to_table
show_savepoints
init_metadata_table
repair_overwrite_hoodie_props
show_metadata_table_partitions
show_logfile_records
downgrade_table
show_clustering
repair_migrate_partition_meta
show_rollbacks
show_logfile_metadata
upgrade_table
repair_add_partition_meta
hive_sync
commits_compare
hdfs_parquet_import
show_commit_write_stats
show_commits
show_archived_commits_metadata
run_compaction
create_savepoint
repair_corrupted_clean_files
stats_wa

具体的定义：

  private def initProcedureBuilders: Map[String, Supplier[ProcedureBuilder]] = {Map((RunCompactionProcedure.NAME, RunCompactionProcedure.builder),(ShowCompactionProcedure.NAME, ShowCompactionProcedure.builder),(CreateSavepointProcedure.NAME, CreateSavepointProcedure.builder),(DeleteSavepointProcedure.NAME, DeleteSavepointProcedure.builder),(RollbackToSavepointProcedure.NAME, RollbackToSavepointProcedure.builder),(RollbackToInstantTimeProcedure.NAME, RollbackToInstantTimeProcedure.builder),(RunClusteringProcedure.NAME, RunClusteringProcedure.builder),(ShowClusteringProcedure.NAME, ShowClusteringProcedure.builder),(ShowCommitsProcedure.NAME, ShowCommitsProcedure.builder),(ShowCommitsMetadataProcedure.NAME, ShowCommitsMetadataProcedure.builder),(ShowArchivedCommitsProcedure.NAME, ShowArchivedCommitsProcedure.builder),(ShowArchivedCommitsMetadataProcedure.NAME, ShowArchivedCommitsMetadataProcedure.builder),(ShowCommitFilesProcedure.NAME, ShowCommitFilesProcedure.builder),(ShowCommitPartitionsProcedure.NAME, ShowCommitPartitionsProcedure.builder),(ShowCommitWriteStatsProcedure.NAME, ShowCommitWriteStatsProcedure.builder),(CommitsCompareProcedure.NAME, CommitsCompareProcedure.builder),(ShowSavepointsProcedure.NAME, ShowSavepointsProcedure.builder),(DeleteMarkerProcedure.NAME, DeleteMarkerProcedure.builder),(ShowRollbacksProcedure.NAME, ShowRollbacksProcedure.builder),(ShowRollbackDetailProcedure.NAME, ShowRollbackDetailProcedure.builder),(ExportInstantsProcedure.NAME, ExportInstantsProcedure.builder),(ShowAllFileSystemViewProcedure.NAME, ShowAllFileSystemViewProcedure.builder),(ShowLatestFileSystemViewProcedure.NAME, ShowLatestFileSystemViewProcedure.builder),(ShowHoodieLogFileMetadataProcedure.NAME, ShowHoodieLogFileMetadataProcedure.builder),(ShowHoodieLogFileRecordsProcedure.NAME, ShowHoodieLogFileRecordsProcedure.builder),(StatsWriteAmplificationProcedure.NAME, StatsWriteAmplificationProcedure.builder),(StatsFileSizeProcedure.NAME, StatsFileSizeProcedure.builder),(HdfsParquetImportProcedure.NAME, HdfsParquetImportProcedure.builder),(RunBootstrapProcedure.NAME, RunBootstrapProcedure.builder),(ShowBootstrapMappingProcedure.NAME, ShowBootstrapMappingProcedure.builder),(ShowBootstrapPartitionsProcedure.NAME, ShowBootstrapPartitionsProcedure.builder),(UpgradeTableProcedure.NAME, UpgradeTableProcedure.builder),(DowngradeTableProcedure.NAME, DowngradeTableProcedure.builder),(ShowMetadataTableFilesProcedure.NAME, ShowMetadataTableFilesProcedure.builder),(ShowMetadataTablePartitionsProcedure.NAME, ShowMetadataTablePartitionsProcedure.builder),(CreateMetadataTableProcedure.NAME, CreateMetadataTableProcedure.builder),(DeleteMetadataTableProcedure.NAME, DeleteMetadataTableProcedure.builder),(InitMetadataTableProcedure.NAME, InitMetadataTableProcedure.builder),(ShowMetadataTableStatsProcedure.NAME, ShowMetadataTableStatsProcedure.builder),(ValidateMetadataTableFilesProcedure.NAME, ValidateMetadataTableFilesProcedure.builder),(ShowFsPathDetailProcedure.NAME, ShowFsPathDetailProcedure.builder),(CopyToTableProcedure.NAME, CopyToTableProcedure.builder),(RepairAddpartitionmetaProcedure.NAME, RepairAddpartitionmetaProcedure.builder),(RepairCorruptedCleanFilesProcedure.NAME, RepairCorruptedCleanFilesProcedure.builder),(RepairDeduplicateProcedure.NAME, RepairDeduplicateProcedure.builder),(RepairMigratePartitionMetaProcedure.NAME, RepairMigratePartitionMetaProcedure.builder),(RepairOverwriteHoodiePropsProcedure.NAME, RepairOverwriteHoodiePropsProcedure.builder),(RunCleanProcedure.NAME, RunCleanProcedure.builder),(ValidateHoodieSyncProcedure.NAME, ValidateHoodieSyncProcedure.builder),(ShowInvalidParquetProcedure.NAME, ShowInvalidParquetProcedure.builder),(HiveSyncProcedure.NAME, HiveSyncProcedure.builder),(CopyToTempView.NAME, CopyToTempView.builder),(ShowCommitExtraMetadataProcedure.NAME, ShowCommitExtraMetadataProcedure.builder),(ShowTablePropertiesProcedure.NAME, ShowTablePropertiesProcedure.builder))

打印代码

initProcedureBuilders.keySet.foreach(println)

建表造数

create table test_hudi_call_cow (id int,name string,price double,ts long,dt string
) using hudipartitioned by (dt)options (primaryKey = 'id',preCombineField = 'ts',type = 'cow');
insert into test_hudi_call_cow values (1,'hudi',10,100,'2021-05-05');
insert into test_hudi_call_cow values (2,'hudi',10,100,'2021-05-05');
insert into test_hudi_call_cow values (3,'hudi',10,100,'2021-05-05');
insert into test_hudi_call_cow values (4,'hudi',10,100,'2021-05-05');

show_table_properties

查看表的properties,以key,value的形式返回hoodie.properties中表的配置

参数

• table 表名
• path 表路径
• limit 可选 默认值10
  table和path两个参数必须得有一个，table的优先级高于path,即如果同时指定table和path，那么以table为准，path不生效。
  输出返回字段：
  key,value

示例

call show_table_properties(table => 'test_hudi_call_cow');
call show_table_properties(path => 'hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow');

默认展示前10条

+--------------------------------------------------+----------------+
|                       key                        |     value      |
+--------------------------------------------------+----------------+
| hoodie.table.precombine.field                    | ts             |
| hoodie.datasource.write.drop.partition.columns   | false          |
| hoodie.table.partition.fields                    | dt             |
| hoodie.table.type                                | COPY_ON_WRITE  |
| hoodie.archivelog.folder                         | archived       |
| hoodie.timeline.layout.version                   | 1              |
| hoodie.table.version                             | 5              |
| hoodie.table.recordkey.fields                    | id             |
| hoodie.table.metadata.partitions                 | files          |
| hoodie.datasource.write.partitionpath.urlencode  | false          |
+--------------------------------------------------+----------------+

可以通过设置limit，将limit值设置大一点，查看所有的配置

call show_table_properties(table => 'test_hudi_call_cow', limit => 100);

+--------------------------------------------------+----------------------------------------------------+
|                       key                        |                       value                        |
+--------------------------------------------------+----------------------------------------------------+
| hoodie.table.precombine.field                    | ts                                                 |
| hoodie.datasource.write.drop.partition.columns   | false                                              |
| hoodie.table.partition.fields                    | dt                                                 |
| hoodie.table.type                                | COPY_ON_WRITE                                      |
| hoodie.archivelog.folder                         | archived                                           |
| hoodie.timeline.layout.version                   | 1                                                  |
| hoodie.table.version                             | 5                                                  |
| hoodie.table.recordkey.fields                    | id                                                 |
| hoodie.table.metadata.partitions                 | files                                              |
| hoodie.datasource.write.partitionpath.urlencode  | false                                              |
| hoodie.database.name                             | hudi                                               |
| hoodie.table.name                                | test_hudi_call_cow                                 |
| hoodie.table.keygenerator.class                  | org.apache.hudi.keygen.SimpleKeyGenerator          |
| hoodie.datasource.write.hive_style_partitioning  | true                                               |
| hoodie.table.create.schema                       | {"type":"record","name":"test_hudi_call_cow_record","namespace":"hoodie.test_hudi_call_cow","fields":[{"name":"_hoodie_commit_time","type":["string","null"]},{"name":"_hoodie_commit_seqno","type":["string","null"]},{"name":"_hoodie_record_key","type":["string","null"]},{"name":"_hoodie_partition_path","type":["string","null"]},{"name":"_hoodie_file_name","type":["string","null"]},{"name":"id","type":["int","null"]},{"name":"name","type":["string","null"]},{"name":"price","type":["double","null"]},{"name":"ts","type":["long","null"]},{"name":"dt","type":["string","null"]}]} |
| hoodie.table.checksum                            | 2721425243                                         |
| hoodie.allow.operation.metadata.field            | false                                              |
+--------------------------------------------------+----------------------------------------------------+

show_commits

参数

• table 表名 必选
• limit 默认值10 可选
  输出返回字段：
  commit_time,action,total_bytes_written,total_files_added,total_files_updated,total_partitions_written,total_records_written,total_update_records_written,total_errors

示例

call show_commits(table => 'test_hudi_call_cow');
call show_commits(table => 'test_hudi_call_cow', limit => 1);

|    commit_time     | action  | total_bytes_written  | total_files_added  | total_files_updated  | total_partitions_written  | total_records_written  | total_update_records_written  | total_errors  |
+--------------------+---------+----------------------+--------------------+----------------------+---------------------------+------------------------+-------------------------------+---------------+
| 20221123205701931  | commit  | 435308               | 0                  | 1                    | 1                         | 4                      | 0                             | 0             |
| 20221123205650038  | commit  | 435279               | 0                  | 1                    | 1                         | 3                      | 0                             | 0             |
| 20221123205636715  | commit  | 435246               | 0                  | 1                    | 1                         | 2                      | 0                             | 0             |
| 20221123205546254  | commit  | 435148               | 1                  | 0                    | 1                         | 1                      | 0                             | 0             |
+--------------------+---------+----------------------+--------------------+----------------------+---------------------------+------------------------+-------------------------------+---------------+

show_commits_metadata

和show_commits功能差不多，不同的是输出字段不一样，和show_commits一样都是通过ShowCommitsProcedures实现的，区别是show_commits_metadata的includeExtraMetadata为true,show_commits的includeExtraMetadata为false

参数

• table 表名 必选
• limit 默认值10 可选
  输出返回字段：
  commit_time,action,partition,file_id,previous_commit,num_writes,num_inserts,num_deletes,num_update_writes,total_errors,total_log_blocks,total_corrupt_log_blocks,total_rollback_blocks,total_log_records, total_updated_records_compacted,total_bytes_written

示例

call show_commits_metadata(table => 'test_hudi_call_cow');
call show_commits_metadata(table => 'test_hudi_call_cow', limit => 1);

+--------------------+---------+----------------+-----------------------------------------+--------------------+-------------+--------------+--------------+--------------------+---------------+-------------------+---------------------------+------------------------+--------------------+----------------------------------+----------------------+
|    commit_time     | action  |   partition    |                 file_id                 |  previous_commit   | num_writes  | num_inserts  | num_deletes  | num_update_writes  | total_errors  | total_log_blocks  | total_corrupt_log_blocks  | total_rollback_blocks  | total_log_records  | total_updated_records_compacted  | total_bytes_written  |
+--------------------+---------+----------------+-----------------------------------------+--------------------+-------------+--------------+--------------+--------------------+---------------+-------------------+---------------------------+------------------------+--------------------+----------------------------------+----------------------+
| 20221123205701931  | commit  | dt=2021-05-05  | 35b07424-6e63-4b65-9182-7c37cbe756b1-0  | 20221123205650038  | 4           | 1            | 0            | 0                  | 0             | 0                 | 0                         | 0                      | 0                  | 0                                | 435308               |
| 20221123205650038  | commit  | dt=2021-05-05  | 35b07424-6e63-4b65-9182-7c37cbe756b1-0  | 20221123205636715  | 3           | 1            | 0            | 0                  | 0             | 0                 | 0                         | 0                      | 0                  | 0                                | 435279               |
| 20221123205636715  | commit  | dt=2021-05-05  | 35b07424-6e63-4b65-9182-7c37cbe756b1-0  | 20221123205546254  | 2           | 1            | 0            | 0                  | 0             | 0                 | 0                         | 0                      | 0                  | 0                                | 435246               |
| 20221123205546254  | commit  | dt=2021-05-05  | 35b07424-6e63-4b65-9182-7c37cbe756b1-0  | null               | 1           | 1            | 0            | 0                  | 0             | 0                 | 0                         | 0                      | 0                  | 0                                | 435148               |
+--------------------+---------+----------------+-----------------------------------------+--------------------+-------------+--------------+--------------+--------------------+---------------+-------------------+---------------------------+------------------------+--------------------+----------------------------------+----------------------+

show_commit_files

根据instantTime返回对应的文件信息，比如fileId

参数

• table 表名 必选
• instant_time 必选
• limit 默认10 可选
  输出返回字段：
  action,partition_path,file_id,previous_commit,total_records_updated,total_records_written,total_bytes_written,total_errors,file_size

示例

call show_commit_files(table => 'test_hudi_call_cow', instant_time => '20221123205701931');

因为测试数据比较少件，且只有一个分区，，所以只有一个文件

+---------+-----------------+-----------------------------------------+--------------------+------------------------+------------------------+----------------------+---------------+------------+
| action  | partition_path  |                 file_id                 |  previous_commit   | total_records_updated  | total_records_written  | total_bytes_written  | total_errors  | file_size  |
+---------+-----------------+-----------------------------------------+--------------------+------------------------+------------------------+----------------------+---------------+------------+
| commit  | dt=2021-05-05   | 35b07424-6e63-4b65-9182-7c37cbe756b1-0  | 20221123205650038  | 0                      | 4                      | 435308               | 0             | 435308     |
+---------+-----------------+-----------------------------------------+--------------------+------------------------+------------------------+----------------------+---------------+------------+

执行下面的sql，使一次commit涉及两个文件

merge into test_hudi_call_cow as t0
using (select 5 as id, 'hudi' as name, 112 as price, 98 as ts, '2022-11-23' as dt,'INSERT' as opt_type unionselect 2 as id, 'hudi_2' as name, 10 as price, 100 as ts, '2021-05-05' as dt,'UPDATE' as opt_type unionselect 4 as id, 'hudi' as name, 10 as price, 100 as ts, '2021-05-05' as dt ,'DELETE' as opt_type) as s0
on t0.id = s0.id
when matched and opt_type!='DELETE' then update set *
when matched and opt_type='DELETE' then delete
when not matched and opt_type!='DELETE' then insert *;

先用show_commits查看最新的commit_time为20221123232449644

call show_commits(table => 'test_hudi_call_cow', limit => 1);+--------------------+---------+----------------------+--------------------+----------------------+---------------------------+------------------------+-------------------------------+---------------+
|    commit_time     | action  | total_bytes_written  | total_files_added  | total_files_updated  | total_partitions_written  | total_records_written  | total_update_records_written  | total_errors  |
+--------------------+---------+----------------------+--------------------+----------------------+---------------------------+------------------------+-------------------------------+---------------+
| 20221123232449644  | commit  | 870474               | 1                  | 1                    | 2                         | 4                      | 1                             | 0             |
+--------------------+---------+----------------------+--------------------+----------------------+---------------------------+------------------------+-------------------------------+---------------+

然后再用show_commit_files看一下20221123232449644对应的文件

call show_commit_files(table => 'test_hudi_call_cow', instant_time => '20221123232449644');+---------+-----------------+-----------------------------------------+--------------------+------------------------+------------------------+----------------------+---------------+------------+
| action  | partition_path  |                 file_id                 |  previous_commit   | total_records_updated  | total_records_written  | total_bytes_written  | total_errors  | file_size  |
+---------+-----------------+-----------------------------------------+--------------------+------------------------+------------------------+----------------------+---------------+------------+
| commit  | dt=2022-11-23   | 8f2aecfd-198f-405b-ab5d-46e0cc997d97-0  | null               | 0                      | 1                      | 435176               | 0             | 435176     |
| commit  | dt=2021-05-05   | 35b07424-6e63-4b65-9182-7c37cbe756b1-0  | 20221123231230786  | 1                      | 3                      | 435298               | 0             | 435298     |
+---------+-----------------+-----------------------------------------+--------------------+------------------------+------------------------+----------------------+---------------+------------+

show_commit_partitions

根据instantTime返回涉及的每个分区对应的文件和记录信息

参数

• table 表名 必选
• instant_time 必选
• limit 默认10 可选
  输出返回字段：
  action,partition_path,total_files_added,total_files_updated,total_records_inserted,total_records_updated,total_bytes_written,total_errors

示例

call show_commit_partitions(table => 'test_hudi_call_cow', instant_time => '20221123232449644');

+---------+-----------------+--------------------+----------------------+-------------------------+------------------------+----------------------+---------------+
| action  | partition_path  | total_files_added  | total_files_updated  | total_records_inserted  | total_records_updated  | total_bytes_written  | total_errors  |
+---------+-----------------+--------------------+----------------------+-------------------------+------------------------+----------------------+---------------+
| commit  | dt=2022-11-23   | 1                  | 0                    | 1                       | 0                      | 435176               | 0             |
| commit  | dt=2021-05-05   | 0                  | 1                    | 0                       | 1                      | 435298               | 0             |
+---------+-----------------+--------------------+----------------------+-------------------------+------------------------+----------------------+---------------+

show_commit_write_stats

根据instantTime返回write_stats

参数：

• table 表名 必选
• instant_time 必选
• limit 默认10 可选
  输出返回字段：
  action,total_bytes_written,total_records_written,avg_record_size

示例

call show_commit_write_stats(table => 'test_hudi_call_cow', instant_time => '20221123232449644');

+---------+----------------------+------------------------+------------------+
| action  | total_bytes_written  | total_records_written  | avg_record_size  |
+---------+----------------------+------------------------+------------------+
| commit  | 870474               | 4                      | 217619           |
+---------+----------------------+------------------------+------------------+

show_commit_extra_metadata

返回.commit、.deltacommit、.replacecommit中的extraMetadata

参数

• table 表名 必选
• instant_time 可选
• limit 默认100 可选
• metadata_key 可选 如schema
  输出返回字段：
  instant_time,action,metadata_key,metadata_value

默认返回最后一个commit文件中的extraMetadata，如果指定了instant_time，那么返回指定instant_time对应的commit文件中的extraMetadata

先看一下.commit里的内容，可以看到里面有一个extraMetadata,并且包含一个key:schema以及schema对应的value

hadoop fs -cat hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/.hoodie/20221123232449644.commit{"partitionToWriteStats" : {"dt=2022-11-23" : [ {"fileId" : "8f2aecfd-198f-405b-ab5d-46e0cc997d97-0","path" : "dt=2022-11-23/8f2aecfd-198f-405b-ab5d-46e0cc997d97-0_1-238-2983_20221123232449644.parquet","cdcStats" : null,"prevCommit" : "null","numWrites" : 1,"numDeletes" : 0,"numUpdateWrites" : 0,"numInserts" : 1,"totalWriteBytes" : 435176,"totalWriteErrors" : 0,"tempPath" : null,"partitionPath" : "dt=2022-11-23","totalLogRecords" : 0,"totalLogFilesCompacted" : 0,"totalLogSizeCompacted" : 0,"totalUpdatedRecordsCompacted" : 0,"totalLogBlocks" : 0,"totalCorruptLogBlock" : 0,"totalRollbackBlocks" : 0,"fileSizeInBytes" : 435176,"minEventTime" : null,"maxEventTime" : null,"runtimeStats" : {"totalScanTime" : 0,"totalUpsertTime" : 0,"totalCreateTime" : 1013}} ],"dt=2021-05-05" : [ {"fileId" : "35b07424-6e63-4b65-9182-7c37cbe756b1-0","path" : "dt=2021-05-05/35b07424-6e63-4b65-9182-7c37cbe756b1-0_0-238-2982_20221123232449644.parquet","cdcStats" : null,"prevCommit" : "20221123231230786","numWrites" : 3,"numDeletes" : 1,"numUpdateWrites" : 1,"numInserts" : 0,"totalWriteBytes" : 435298,"totalWriteErrors" : 0,"tempPath" : null,"partitionPath" : "dt=2021-05-05","totalLogRecords" : 0,"totalLogFilesCompacted" : 0,"totalLogSizeCompacted" : 0,"totalUpdatedRecordsCompacted" : 0,"totalLogBlocks" : 0,"totalCorruptLogBlock" : 0,"totalRollbackBlocks" : 0,"fileSizeInBytes" : 435298,"minEventTime" : null,"maxEventTime" : null,"runtimeStats" : {"totalScanTime" : 0,"totalUpsertTime" : 4162,"totalCreateTime" : 0}} ]},"compacted" : false,"extraMetadata" : {"schema" : "{\"type\":\"record\",\"name\":\"test_hudi_call_cow_record\",\"namespace\":\"hoodie.test_hudi_call_cow\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"price\",\"type\":\"double\"},{\"name\":\"ts\",\"type\":\"long\"},{\"name\":\"dt\",\"type\":\"string\"}]}"},"operationType" : "UPSERT"
}

示例

call show_commit_extra_metadata(table => 'test_hudi_call_cow');

+--------------------+---------+---------------+----------------------------------------------------+
|    instant_time    | action  | metadata_key  |                   metadata_value                   |
+--------------------+---------+---------------+----------------------------------------------------+
| 20221123232449644  | commit  | schema        | {"type":"record","name":"test_hudi_call_cow_record","namespace":"hoodie.test_hudi_call_cow","fields":[{"name":"id","type":"int"},{"name":"name","type":"string"},{"name":"price","type":"double"},{"name":"ts","type":"long"},{"name":"dt","type":"string"}]} |
+--------------------+---------+---------------+----------------------------------------------------+

call show_commit_extra_metadata(table => 'test_hudi_call_cow', instant_time => '20221123205701931', metadata_key => 'schema', limit => 10);

+--------------------+---------+---------------+----------------------------------------------------+
|    instant_time    | action  | metadata_key  |                   metadata_value                   |
+--------------------+---------+---------------+----------------------------------------------------+
| 20221123205701931  | commit  | schema        | {"type":"record","name":"test_hudi_call_cow_record","namespace":"hoodie.test_hudi_call_cow","fields":[{"name":"id","type":"int"},{"name":"name","type":"string"},{"name":"price","type":"double"},{"name":"ts","type":"long"},{"name":"dt","type":"string"}]} |
+--------------------+---------+---------------+----------------------------------------------------+

我目前已知的extraMetadata中只有schema，且只有一条，所以只返回一条记录，且指定不指定metadata_key效果一样，不确定是否还有其他的extraMetadata

show_fs_path_detail

展示指定路径下面的文件和路径的统计信息，默认按照文件大小进行排序
返回.commit、.deltacommit、.replacecommit中的extraMetadata

参数

• path 表文件路径 必选
• is_sub 可选 是否查询子目录,只查询一级子目录 默认false
• sort 可选 是否按文件大小排序 默认true
• metadata_key 可选 如schema
  输出返回字段：
  instant_time,action,metadata_key,metadata_value

示例

call show_fs_path_detail(path => 'hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow');+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+
| path_num  | file_num  | storage_size  | storage_size(unit)  |                    storage_path                    | space_consumed  |  quota   | space_quota  |
+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+
| 22        | 53        | 3200109       | 3.05MB              | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow | -1              | 9600327  | -1           |
+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+

查询一级子目录

call show_fs_path_detail(path => 'hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow', is_sub => true);+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+
| path_num  | file_num  | storage_size  | storage_size(unit)  |                    storage_path                    | space_consumed  |  quota   | space_quota  |
+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+
| 1         | 7         | 2611728       | 2.49MB              | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05 | -1              | 7835184  | -1           |
| 1         | 2         | 435272        | 425.07KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2022-11-23 | -1              | 1305816  | -1           |
| 19        | 44        | 153109        | 149.52KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/.hoodie | -1              | 459327   | -1           |
+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+

查询二级子目录

call show_fs_path_detail(path => 'hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05', is_sub => true);+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+
| path_num  | file_num  | storage_size  | storage_size(unit)  |                    storage_path                    | space_consumed  |  quota   | space_quota  |
+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+
| 0         | 1         | 435353        | 425.15KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05/35b07424-6e63-4b65-9182-7c37cbe756b1-0_0-192-1549_20221123231230786.parquet | -1              | 1306059  | -1           |
| 0         | 1         | 435308        | 425.11KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05/35b07424-6e63-4b65-9182-7c37cbe756b1-0_0-147-118_20221123205701931.parquet | -1              | 1305924  | -1           |
| 0         | 1         | 435298        | 425.10KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05/35b07424-6e63-4b65-9182-7c37cbe756b1-0_0-238-2982_20221123232449644.parquet | -1              | 1305894  | -1           |
| 0         | 1         | 435279        | 425.08KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05/35b07424-6e63-4b65-9182-7c37cbe756b1-0_0-105-83_20221123205650038.parquet | -1              | 1305837  | -1           |
| 0         | 1         | 435246        | 425.04KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05/35b07424-6e63-4b65-9182-7c37cbe756b1-0_0-66-52_20221123205636715.parquet | -1              | 1305738  | -1           |
| 0         | 1         | 435148        | 424.95KB            | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05/35b07424-6e63-4b65-9182-7c37cbe756b1-0_0-27-21_20221123205546254.parquet | -1              | 1305444  | -1           |
| 0         | 1         | 96            | 96B                 | hdfs://cluster1/warehouse/tablespace/managed/hive/hudi.db/test_hudi_call_cow/dt=2021-05-05/.hoodie_partition_metadata | -1              | 288      | -1           |
+-----------+-----------+---------------+---------------------+----------------------------------------------------+-----------------+----------+--------------+

相关阅读

• Apache Hudi 入门学习总结
• Hudi Spark SQL总结
• Spark3.12+Kyuubi1.5.2+kyuubi-spark-authz源码编译打包+部署配置HA
• Hudi Spark SQL源码学习总结-Create Table
• Hudi Spark SQL源码学习总结-CTAS
• Hudi Spark源码学习总结-df.write.format(“hudi”).save
• Hudi Spark源码学习总结-spark.read.format(“hudi”).load
• Hudi Spark源码学习总结-spark.read.format(“hudi”).load（2）
• Hudi Spark SQL源码学习总结-select（查询）
• Hudi源码 | Insert源码分析总结（二）(WorkloadProfile)
• 开源经验分享 | 如何从一名小白成为Apache Hudi Contributor