Flink State 状态后端分析
创始人
2024-05-27 16:52:30
0

flink状态实现分析

state

 *             State*               |*               +-------------------InternalKvState*               |                         |*          MergingState                   |*               |                         |*               +-----------------InternalMergingState*               |                         |*      +--------+------+                  |*      |               |                  |* ReducingState    ListState        +-----+-----------------+*      |               |            |                       |*      +-----------+   +-----------   -----------------InternalListState*                  |                |*                  +---------InternalReducingState

MemoryState

AbstractHeapStateHeapMapStateInternalMapStateInternalKvStateStateAbstractHeapMergingStateHeapListStateInternalListStateAbstractHeapAppendingStateInternalMergingStateInternalAppendingStateHeapValueStateInternalValueState

RocksDBState

StateInternalKvStateAbstractRocksDBStateRocksDBMapStateRocksDBListStateRocksDBValueStateRocksDBReducingStateRocksDBAggregatingState
class RocksDBMapState extends AbstractRocksDBState> {private TypeSerializer userKeySerializer;private TypeSerializer userValueSerializer;private RocksDBMapState(ColumnFamilyHandle columnFamily,TypeSerializer namespaceSerializer,TypeSerializer> valueSerializer,Map defaultValue,RocksDBKeyedStateBackend backend);public TypeSerializer getKeySerializer();public TypeSerializer getNamespaceSerializer();public TypeSerializer> getValueSerializer();public UV get(UK userKey){ //直接读rocksdbbyte[] rawKeyBytes =serializeCurrentKeyWithGroupAndNamespacePlusUserKey(userKey, userKeySerializer);byte[] rawValueBytes = backend.db.get(columnFamily, rawKeyBytes);return (rawValueBytes == null? null: deserializeUserValue(dataInputView, rawValueBytes, userValueSerializer));}public void put(UK userKey, UV userValue){ //直接写rocksdbbyte[] rawKeyBytes =serializeCurrentKeyWithGroupAndNamespacePlusUserKey(userKey, userKeySerializer);byte[] rawValueBytes = serializeValueNullSensitive(userValue, userValueSerializer);backend.db.put(columnFamily, writeOptions, rawKeyBytes, rawValueBytes); //backend.db是RocksDBKeyedStateBackend}public void putAll(Map map);public void remove(UK userKey);public boolean contains(UK userKey);public Iterable> entries();public Iterable keys();public Iterable values();public boolean isEmpty();public void clear();static  IS create(StateDescriptor stateDesc,Tuple2>registerResult,RocksDBKeyedStateBackend backend) { //backend在这里传入return (IS)new RocksDBMapState<>(registerResult.f0,registerResult.f1.getNamespaceSerializer(),(TypeSerializer>) registerResult.f1.getStateSerializer(),(Map) stateDesc.getDefaultValue(),backend);}
}

backend与checkpoint

AbstractKeyedStateBackendRocksDBKeyedStateBackendCheckpointableKeyedStateBackendKeyedStateBackendSnapshotableHeapKeyedStateBackendOperatorStateBackendDefaultOperatorStateBackendOperatorStateStore
public interface Snapshotable {RunnableFuture snapshot(long checkpointId,long timestamp,@Nonnull CheckpointStreamFactory streamFactory,@Nonnull CheckpointOptions checkpointOptions)throws Exception;
}

FSBackend

  • FsStateBackend中createKeyedStateBackend是创建了HeapKeyedStateBackend
  • FsStateBackend中createOperatorStateBackend是创建了DefaultOperatorStateBackend
  • DefaultOperatorStateBackend创建了PartitionableListState, 是State的子类
AbstractFileStateBackendFsStateBackendAbstractStateBackendCheckpointStorageStateBackendConfigurableStateBackend
public interface StateBackend extends java.io.Serializable {default String getName() {return this.getClass().getSimpleName();} CheckpointableKeyedStateBackend createKeyedStateBackend(Environment env,JobID jobID,String operatorIdentifier,TypeSerializer keySerializer,int numberOfKeyGroups,KeyGroupRange keyGroupRange,TaskKvStateRegistry kvStateRegistry,TtlTimeProvider ttlTimeProvider,MetricGroup metricGroup,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws Exception;OperatorStateBackend createOperatorStateBackend(Environment env,String operatorIdentifier,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws Exception;/** Whether the state backend uses Flink's managed memory. */default boolean useManagedMemory() {return false;}}
public class FsStateBackend extends AbstractFileStateBackend implements ConfigurableStateBackend {public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {checkNotNull(jobId, "jobId");return new FsCheckpointStorageAccess(getCheckpointPath(),getSavepointPath(),jobId,getMinFileSizeThreshold(),getWriteBufferSize());}public  AbstractKeyedStateBackend createKeyedStateBackend(Environment env,JobID jobID,String operatorIdentifier,TypeSerializer keySerializer,int numberOfKeyGroups,KeyGroupRange keyGroupRange,TaskKvStateRegistry kvStateRegistry,TtlTimeProvider ttlTimeProvider,MetricGroup metricGroup,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws BackendBuildingException {TaskStateManager taskStateManager = env.getTaskStateManager();LocalRecoveryConfig localRecoveryConfig = taskStateManager.createLocalRecoveryConfig();HeapPriorityQueueSetFactory priorityQueueSetFactory =new HeapPriorityQueueSetFactory(keyGroupRange, numberOfKeyGroups, 128);LatencyTrackingStateConfig latencyTrackingStateConfig =latencyTrackingConfigBuilder.setMetricGroup(metricGroup).build();return new HeapKeyedStateBackendBuilder<>( //这里是HeapKeyedStateBackendBuilderkvStateRegistry,keySerializer,env.getUserCodeClassLoader().asClassLoader(),numberOfKeyGroups,keyGroupRange,env.getExecutionConfig(),ttlTimeProvider,latencyTrackingStateConfig,stateHandles,AbstractStateBackend.getCompressionDecorator(env.getExecutionConfig()),localRecoveryConfig,priorityQueueSetFactory,isUsingAsynchronousSnapshots(),cancelStreamRegistry).build();}@Overridepublic OperatorStateBackend createOperatorStateBackend(Environment env,String operatorIdentifier,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws BackendBuildingException {return new DefaultOperatorStateBackendBuilder(  //这里是DefaultOperatorStateBackendBuilderenv.getUserCodeClassLoader().asClassLoader(),env.getExecutionConfig(),isUsingAsynchronousSnapshots(),stateHandles,cancelStreamRegistry).build();}
}

memory backend

  • MemoryStateBackend中createOperatorStateBackend是创建了DefaultOperatorStateBackend
  • MemoryStateBackend中createKeyedStateBackend是创建了HeapKeyedStateBackendBackend
  • 最终调用了HeapMapState::Create创建state
AbstractFileStateBackendMemoryStateBackendConfigurableStateBackendAbstractStateBackendCheckpointStorageStateBackend

flink checkpoint

CheckpointStorage+resolveCheckpoint(String externalPointer)+createCheckpointStorage(JobID jobId)RocksDBStateBackend+checkpointStreamBackend : StateBackendCheckpointStorageAccessAbstractFsCheckpointStorageAccessFsCheckpointStorageAccessMemoryBackendCheckpointStorageAccess RestoreOperationRocksDBRestoreOperationRocksDBFullRestoreOperationRocksDBHeapTimersFullRestoreOperationRocksDBIncrementalRestoreOperationRocksDBSnapshotOperationRocksDBIncrementalSnapshotOperationRocksDBNativeFullSnapshotOperation

参考资料

https://www.jianshu.com/p/569a7e67c1b3
https://blog.csdn.net/u010942041/article/details/114944767
https://cloud.tencent.com/developer/article/1792720
https://blog.51cto.com/dataclub/5351042
https://www.cnblogs.com/lighten/p/13234350.html
https://cloud.tencent.com/developer/article/1765572
https://blog.csdn.net/m0_63475429/article/details/127417649
https://blog.csdn.net/Direction_Wind/article/details/125646616

相关内容

热门资讯

乌兰的风吹到了湖州 转自:湖州日报  7月4日,2025年“湖乌情缘·光影之旅”湖州&乌兰主题摄影巡展在南太湖新区月亮广...
南京公积金异地贷款“扩圈”全省... 南报网讯(记者 孙琳 实习生 胡洁 蒋丹) 南京公积金异地贷款“扩圈”全省首笔落地!7月5日上午,专...
旅客突然跳入股道致G7545次... 上海铁路局杭州站官方微博@铁路杭州站 7月5日发文称,当天16时34分许,G7545次列车驶入杭州东...
马斯克宣布“美国党”成立! 当地时间7月5日,美国企业家埃隆·马斯克在社交媒体平台X上发文称,“美国党”于当日成立,以还给人民自...
前5月“洪城科贷通”实际放贷超...   本报讯(洪观新闻记者 黄之昊)7月5日,记者从市科技局获悉,今年1月至5月,南昌市“洪城科贷通”...
市监委举行宪法宣誓仪式   本报讯(刘希曦 洪观新闻记者 吴潇远)7月4日,市监察委员会举行市监委副主任和市监委委员宪法宣誓...
北京亦庄启动打造6G创新发展先... “北京亦庄”微信公众号消息,在7月5日举办的2025全球数字经济大会“6G技术创新与产业发展峰会”上...
缓解道路交通压力 提升区域通行...   本报讯(洪观新闻记者 万能)7月5日,省委常委、南昌市委书记、赣江新区党工委书记李红军深入一线,...
马来西亚称中俄同意签署《东南亚... (转自:上林下夕)外交部发言人毛宁主持7月3日例行记者会。会上有外媒记者提问称:马来西亚官方媒体昨日...
早报特朗普称8月1日起实施新关... 转自:财联社宏观新闻1、中共中央政治局委员、国务院副总理刘国中3日至5日到广东调研。他强调,要深入学...