Flink State 状态后端分析
创始人
2024-05-27 16:52:30
0

flink状态实现分析

state

 *             State*               |*               +-------------------InternalKvState*               |                         |*          MergingState                   |*               |                         |*               +-----------------InternalMergingState*               |                         |*      +--------+------+                  |*      |               |                  |* ReducingState    ListState        +-----+-----------------+*      |               |            |                       |*      +-----------+   +-----------   -----------------InternalListState*                  |                |*                  +---------InternalReducingState

MemoryState

AbstractHeapStateHeapMapStateInternalMapStateInternalKvStateStateAbstractHeapMergingStateHeapListStateInternalListStateAbstractHeapAppendingStateInternalMergingStateInternalAppendingStateHeapValueStateInternalValueState

RocksDBState

StateInternalKvStateAbstractRocksDBStateRocksDBMapStateRocksDBListStateRocksDBValueStateRocksDBReducingStateRocksDBAggregatingState
class RocksDBMapState extends AbstractRocksDBState> {private TypeSerializer userKeySerializer;private TypeSerializer userValueSerializer;private RocksDBMapState(ColumnFamilyHandle columnFamily,TypeSerializer namespaceSerializer,TypeSerializer> valueSerializer,Map defaultValue,RocksDBKeyedStateBackend backend);public TypeSerializer getKeySerializer();public TypeSerializer getNamespaceSerializer();public TypeSerializer> getValueSerializer();public UV get(UK userKey){ //直接读rocksdbbyte[] rawKeyBytes =serializeCurrentKeyWithGroupAndNamespacePlusUserKey(userKey, userKeySerializer);byte[] rawValueBytes = backend.db.get(columnFamily, rawKeyBytes);return (rawValueBytes == null? null: deserializeUserValue(dataInputView, rawValueBytes, userValueSerializer));}public void put(UK userKey, UV userValue){ //直接写rocksdbbyte[] rawKeyBytes =serializeCurrentKeyWithGroupAndNamespacePlusUserKey(userKey, userKeySerializer);byte[] rawValueBytes = serializeValueNullSensitive(userValue, userValueSerializer);backend.db.put(columnFamily, writeOptions, rawKeyBytes, rawValueBytes); //backend.db是RocksDBKeyedStateBackend}public void putAll(Map map);public void remove(UK userKey);public boolean contains(UK userKey);public Iterable> entries();public Iterable keys();public Iterable values();public boolean isEmpty();public void clear();static  IS create(StateDescriptor stateDesc,Tuple2>registerResult,RocksDBKeyedStateBackend backend) { //backend在这里传入return (IS)new RocksDBMapState<>(registerResult.f0,registerResult.f1.getNamespaceSerializer(),(TypeSerializer>) registerResult.f1.getStateSerializer(),(Map) stateDesc.getDefaultValue(),backend);}
}

backend与checkpoint

AbstractKeyedStateBackendRocksDBKeyedStateBackendCheckpointableKeyedStateBackendKeyedStateBackendSnapshotableHeapKeyedStateBackendOperatorStateBackendDefaultOperatorStateBackendOperatorStateStore
public interface Snapshotable {RunnableFuture snapshot(long checkpointId,long timestamp,@Nonnull CheckpointStreamFactory streamFactory,@Nonnull CheckpointOptions checkpointOptions)throws Exception;
}

FSBackend

  • FsStateBackend中createKeyedStateBackend是创建了HeapKeyedStateBackend
  • FsStateBackend中createOperatorStateBackend是创建了DefaultOperatorStateBackend
  • DefaultOperatorStateBackend创建了PartitionableListState, 是State的子类
AbstractFileStateBackendFsStateBackendAbstractStateBackendCheckpointStorageStateBackendConfigurableStateBackend
public interface StateBackend extends java.io.Serializable {default String getName() {return this.getClass().getSimpleName();} CheckpointableKeyedStateBackend createKeyedStateBackend(Environment env,JobID jobID,String operatorIdentifier,TypeSerializer keySerializer,int numberOfKeyGroups,KeyGroupRange keyGroupRange,TaskKvStateRegistry kvStateRegistry,TtlTimeProvider ttlTimeProvider,MetricGroup metricGroup,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws Exception;OperatorStateBackend createOperatorStateBackend(Environment env,String operatorIdentifier,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws Exception;/** Whether the state backend uses Flink's managed memory. */default boolean useManagedMemory() {return false;}}
public class FsStateBackend extends AbstractFileStateBackend implements ConfigurableStateBackend {public CheckpointStorageAccess createCheckpointStorage(JobID jobId) throws IOException {checkNotNull(jobId, "jobId");return new FsCheckpointStorageAccess(getCheckpointPath(),getSavepointPath(),jobId,getMinFileSizeThreshold(),getWriteBufferSize());}public  AbstractKeyedStateBackend createKeyedStateBackend(Environment env,JobID jobID,String operatorIdentifier,TypeSerializer keySerializer,int numberOfKeyGroups,KeyGroupRange keyGroupRange,TaskKvStateRegistry kvStateRegistry,TtlTimeProvider ttlTimeProvider,MetricGroup metricGroup,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws BackendBuildingException {TaskStateManager taskStateManager = env.getTaskStateManager();LocalRecoveryConfig localRecoveryConfig = taskStateManager.createLocalRecoveryConfig();HeapPriorityQueueSetFactory priorityQueueSetFactory =new HeapPriorityQueueSetFactory(keyGroupRange, numberOfKeyGroups, 128);LatencyTrackingStateConfig latencyTrackingStateConfig =latencyTrackingConfigBuilder.setMetricGroup(metricGroup).build();return new HeapKeyedStateBackendBuilder<>( //这里是HeapKeyedStateBackendBuilderkvStateRegistry,keySerializer,env.getUserCodeClassLoader().asClassLoader(),numberOfKeyGroups,keyGroupRange,env.getExecutionConfig(),ttlTimeProvider,latencyTrackingStateConfig,stateHandles,AbstractStateBackend.getCompressionDecorator(env.getExecutionConfig()),localRecoveryConfig,priorityQueueSetFactory,isUsingAsynchronousSnapshots(),cancelStreamRegistry).build();}@Overridepublic OperatorStateBackend createOperatorStateBackend(Environment env,String operatorIdentifier,@Nonnull Collection stateHandles,CloseableRegistry cancelStreamRegistry)throws BackendBuildingException {return new DefaultOperatorStateBackendBuilder(  //这里是DefaultOperatorStateBackendBuilderenv.getUserCodeClassLoader().asClassLoader(),env.getExecutionConfig(),isUsingAsynchronousSnapshots(),stateHandles,cancelStreamRegistry).build();}
}

memory backend

  • MemoryStateBackend中createOperatorStateBackend是创建了DefaultOperatorStateBackend
  • MemoryStateBackend中createKeyedStateBackend是创建了HeapKeyedStateBackendBackend
  • 最终调用了HeapMapState::Create创建state
AbstractFileStateBackendMemoryStateBackendConfigurableStateBackendAbstractStateBackendCheckpointStorageStateBackend

flink checkpoint

CheckpointStorage+resolveCheckpoint(String externalPointer)+createCheckpointStorage(JobID jobId)RocksDBStateBackend+checkpointStreamBackend : StateBackendCheckpointStorageAccessAbstractFsCheckpointStorageAccessFsCheckpointStorageAccessMemoryBackendCheckpointStorageAccess RestoreOperationRocksDBRestoreOperationRocksDBFullRestoreOperationRocksDBHeapTimersFullRestoreOperationRocksDBIncrementalRestoreOperationRocksDBSnapshotOperationRocksDBIncrementalSnapshotOperationRocksDBNativeFullSnapshotOperation

参考资料

https://www.jianshu.com/p/569a7e67c1b3
https://blog.csdn.net/u010942041/article/details/114944767
https://cloud.tencent.com/developer/article/1792720
https://blog.51cto.com/dataclub/5351042
https://www.cnblogs.com/lighten/p/13234350.html
https://cloud.tencent.com/developer/article/1765572
https://blog.csdn.net/m0_63475429/article/details/127417649
https://blog.csdn.net/Direction_Wind/article/details/125646616

相关内容

热门资讯

关于一个童话 关于一个童话这个童话叫"田螺姑娘"那个公主?她不是公主~是个田螺精~但是后来他们一样过得很幸福田螺姑...
办理遗产买卖房产时产生欠条有法... 办理遗产买卖房产时产生欠条有法律效应吗?请详细描述,我们好准确回答.楼主所问的问题太简单,建议具体叙...
昆明好耍的地方 昆明好耍的地方1、大渔公园大渔公园位于昆明市呈贡县渔浦路怡和小区北,是集娱乐休闲、旅游度假为一体的好...
宋末词四大家 宋末词四大家只说名字周邦彦辛弃疾王沂孙吴文英张炎、王沂孙、蒋捷、周密宋词四大家是周邦彦、辛弃疾、王沂...
求:电影《布达佩斯之恋》里面钢... 求:电影《布达佩斯之恋》里面钢琴家安德拉许去餐馆应聘时弹的钢琴曲叫什么?不是那个黑色星期天!真是太好...
与张柏芝频传恋情,吴建豪和张柏... 与张柏芝频传恋情,吴建豪和张柏芝是否真的恋爱了?他们是真的没有恋爱,因为张柏芝还有谢霆锋的几个孩子,...
极品都市小说求推荐 百万字以上... 极品都市小说求推荐 百万字以上最好最强弃少,这个比较好看
什么是“状元、榜眼、探花”是何... 什么是“状元、榜眼、探花”是何典故?人文常识:古代科举考试中的第三名怎么称呼?状元、榜眼还是探花状元...
网上不是说冥王星在2010年被... 网上不是说冥王星在2010年被重新归为九大行星之列了吗?太阳系八大行星,不包括冥王星,如果冥王星算进...
韩版恶作剧之吻其他演员 韩版恶作剧之吻其他演员貌似确定了,是金贤重朴宝英男主是金贤重 女主的话 朴宝英可能性更大一些...
约会大作战第一季,第二季分别有... 约会大作战第一季,第二季分别有多少集第一季12急 两集ova 一集剧场版 第二季10集第一集十二集,...
关于母子间误会的故事 关于母子间误会的故事急求一个故事,关于母亲为孩子无悔付出,却在不经意间伤害了孩子,导致孩子误会了母亲...
一部电视剧是关于选美的几个女孩... 一部电视剧是关于选美的几个女孩子的,香港的,片子很老了,忘记叫什么名字了,诸位可否帮帮忙啊?香港小姐...
你听过最好的情话有哪些? 你听过最好的情话有哪些?鲜花给你,戒指给你,茱萸给你,柔情似水给你,热情如火给你,夏天的西瓜给你,秋...
倒影什么? 倒影什么?倒影是光照射在平静的水面上所成的等大虚像。成像原理遵循平面镜成像的原理。
益安宁丸的功效有什么? 益安宁丸的功效有什么?养心安神,健脾益肝,补血活血,还可以治疗肝肾不足,气血虚弱,所引起的腰漆酸软,...
求《声声慢》小石头和孩子们 求《声声慢》小石头和孩子们求《声声慢》小石头和孩子们声声慢,寻寻觅觅冷冷清清凄凄啴啴七七嗯这一个铲子...
我的同桌 不要抄袭。急。 我的同桌 不要抄袭。急。什么哦?作文还是?怎一个“慢”字了得  我有一个同桌,她是个文文静静的女孩子...
文风类似疯丢子的作者有哪些? 文风类似疯丢子的作者有哪些?文风类似疯丢子的作者有哪些?最好推荐他们的小说。《银河第一纪元》迷路的龙
紫薇适合在家里养吗 紫薇适合在家里养吗紫薇属于乔木,树形太大,养在院子里应该可以的,如果养在室内可能不适宜,很难养好。