Kubernetes client-go DeltaFIFO 源码分析

互联网 2021/9/30 17:12:17

<!DOCTYPE html><html lang="en"><head> <script async src="https://www.googletagmanager.com/gtag/js?id=G-994J9VCT2Q"></script><script> window.dataLayer = window.dataLayer || []; function gtag()…

<!DOCTYPE html>
<html lang="en">
<head>


<script async src="https://www.googletagmanager.com/gtag/js?id=G-994J9VCT2Q"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());

gtag('config', 'G-994J9VCT2Q');
</script>

<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Kubernetes client-go DeltaFIFO 源码分析 - Daniel Hu&#39;s Blog | 胡涛的博客</title>
<meta name="renderer" content="webkit" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"/>

<meta http-equiv="Cache-Control" content="no-transform" />
<meta http-equiv="Cache-Control" content="no-siteapp" />

<meta name="theme-color" content="#f8f5ec" />
<meta name="msapplication-navbutton-color" content="#f8f5ec">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="#f8f5ec">


<meta name="author" content="Daniel Hu" /><meta name="description" content="概述 源码版本信息 Project: kubernetes Branch: master Last commit id: d25d741c Date: 2021-09-26 DeltaFIFO 是自定义控制器开发涉及到的一个重要组件,今天我们来详细研究下 client-go 里 DeltaFIFO 相关代码。 Queue 接口 类似 workqueue 里的队列概念," /><meta name="keywords" content="k8s, kubernetes, docker, istio, 微服务, 容器" />

 

 


<meta name="generator" content="Hugo 0.88.1 with theme even" />


<link rel="canonical" href="https://danielhu.cn/post/k8s/client-go-deltafifo/" />
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/manifest.json">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">

<script data-ad-client="ca-pub-5054040294198003" async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>

 

<link href="/sass/main.min.b5a744db6de49a86cadafb3b70f555ab443f83c307a483402259e94726b045ff.css" rel="stylesheet">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@fancyapps/fancybox@3.1.20/dist/jquery.fancybox.min.css" integrity="sha256-7TyXnr2YU040zfSP+rEcz29ggW4j56/ujTPwjMzyqFY=" crossorigin="anonymous">


<meta property="og:title" content="Kubernetes client-go DeltaFIFO 源码分析" />
<meta property="og:description" content="概述 源码版本信息 Project: kubernetes Branch: master Last commit id: d25d741c Date: 2021-09-26 DeltaFIFO 是自定义控制器开发涉及到的一个重要组件,今天我们来详细研究下 client-go 里 DeltaFIFO 相关代码。 Queue 接口 类似 workqueue 里的队列概念," />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://danielhu.cn/post/k8s/client-go-deltafifo/" /><meta property="article:section" content="post" />
<meta property="article:published_time" content="2021-09-28T07:57:48+08:00" />
<meta property="article:modified_time" content="2021-09-29T17:06:33+08:00" />

<meta itemprop="name" content="Kubernetes client-go DeltaFIFO 源码分析">
<meta itemprop="description" content="概述 源码版本信息 Project: kubernetes Branch: master Last commit id: d25d741c Date: 2021-09-26 DeltaFIFO 是自定义控制器开发涉及到的一个重要组件,今天我们来详细研究下 client-go 里 DeltaFIFO 相关代码。 Queue 接口 类似 workqueue 里的队列概念,"><meta itemprop="datePublished" content="2021-09-28T07:57:48+08:00" />
<meta itemprop="dateModified" content="2021-09-29T17:06:33+08:00" />
<meta itemprop="wordCount" content="2942">
<meta itemprop="keywords" content="k8s,client-go,deltafifo,源码分析," /><meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Kubernetes client-go DeltaFIFO 源码分析"/>
<meta name="twitter:description" content="概述 源码版本信息 Project: kubernetes Branch: master Last commit id: d25d741c Date: 2021-09-26 DeltaFIFO 是自定义控制器开发涉及到的一个重要组件,今天我们来详细研究下 client-go 里 DeltaFIFO 相关代码。 Queue 接口 类似 workqueue 里的队列概念,"/>

<!--[if lte IE 9]>
<script src="https://cdnjs.cloudflare.com/ajax/libs/classlist/1.1.20170427/classList.min.js"></script>
<![endif]-->

<!--[if lt IE 9]>
<script src="https://cdn.jsdelivr.net/npm/html5shiv@3.7.3/dist/html5shiv.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/respond.js@1.4.2/dest/respond.min.js"></script>
<![endif]-->

</head>
<body>
<div id="mobile-navbar" class="mobile-navbar">
<div class="mobile-header-logo">
<a href="/" class="logo">Daniel Hu&#39;s Blog</a>
</div>
<div class="mobile-navbar-icon">
<span></span>
<span></span>
<span></span>
</div>
</div>
<nav id="mobile-menu" class="mobile-menu slideout-menu">
<ul class="mobile-menu-list">
<a href="/post/">
<li class="mobile-menu-item">Archives(目录)</li>
</a><a href="/tags/">
<li class="mobile-menu-item">Tags</li>
</a><a href="/about/">
<li class="mobile-menu-item">About</li>
</a><a href="/k8s-source-code-analysis/">
<li class="mobile-menu-item">E-book</li>
</a>
</ul>

 


</nav>

<div class="container" id="mobile-panel">
<header id="header" class="header">
<div class="logo-wrapper">
<a href="/" class="logo">Daniel Hu&#39;s Blog</a>
</div>

 

 

<nav class="site-navbar">
<ul id="menu" class="menu">
<li class="menu-item">
<a class="menu-item-link" href="/post/">Archives(目录)</a>
</li><li class="menu-item">
<a class="menu-item-link" href="/tags/">Tags</a>
</li><li class="menu-item">
<a class="menu-item-link" href="/about/">About</a>
</li><li class="menu-item">
<a class="menu-item-link" href="/k8s-source-code-analysis/">E-book</a>
</li>
</ul>
</nav>

</header>

<main id="main" class="main">
<div class="content-wrapper">
<div id="content" class="content">
<article class="post">

<header class="post-header">
<h1 class="post-title">Kubernetes client-go DeltaFIFO 源码分析</h1>

<div class="post-meta">
<span class="post-time"> 2021-09-28 </span>

<span class="more-meta"> 2942 words </span>
<span class="more-meta"> 6 mins read </span>

</div>
</header>

<div class="post-toc" id="post-toc">
<h2 class="post-toc-title">Contents</h2>
<div class="post-toc-content">
<nav id="TableOfContents">
<ul>
<li>
<ul>
<li><a href="#概述">概述</a></li>
<li><a href="#queue-接口">Queue 接口</a></li>
<li><a href="#deltafifo">DeltaFIFO</a></li>
<li><a href="#添加元素---queueactionlocked">添加元素 - queueActionLocked()</a></li>
<li><a href="#pop">Pop()</a></li>
<li><a href="#replace">Replace()</a></li>
</ul>
</li>
</ul>
</nav>
</div>
</div>
<div class="post-content">
<h2 id="概述">概述</h2>
<blockquote>
<p>源码版本信息</p>
<ul>
<li>Project: kubernetes</li>
<li>Branch: master</li>
<li>Last commit id: d25d741c</li>
<li>Date: 2021-09-26</li>
</ul>
</blockquote>
<p>DeltaFIFO 是自定义控制器开发涉及到的一个重要组件,今天我们来详细研究下 client-go 里 DeltaFIFO 相关代码。</p>
<p><img src="https://gitee.com/daniel-hutao/images/raw/master/delta_fifo.png" alt=""></p>
<h2 id="queue-接口">Queue 接口</h2>
<p>类似 workqueue 里的队列概念,这里也有一个队列,Queue 接口定义在 <em>client-go/tools/cache</em> 包中的 <em>fifo.go</em> 文件里,看下有哪些方法:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">Queue</span> <span class="kd">interface</span> <span class="p">{</span>
<span class="nx">Store</span>
<span class="nf">Pop</span><span class="p">(</span><span class="nx">PopProcessFunc</span><span class="p">)</span> <span class="p">(</span><span class="kd">interface</span><span class="p">{},</span> <span class="kt">error</span><span class="p">)</span> <span class="c1">// 会阻塞,知道有一个元素可以被 pop 出来,或者队列关闭
</span><span class="c1"></span> <span class="nf">AddIfNotPresent</span><span class="p">(</span><span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span>
<span class="nf">HasSynced</span><span class="p">()</span> <span class="kt">bool</span>
<span class="nf">Close</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>这里嵌里一个 Store 接口,对应定义如下:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">Store</span> <span class="kd">interface</span> <span class="p">{</span>
<span class="nf">Add</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span>
<span class="nf">Update</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span>
<span class="nf">Delete</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span>
<span class="nf">List</span><span class="p">()</span> <span class="p">[]</span><span class="kd">interface</span><span class="p">{}</span>
<span class="nf">ListKeys</span><span class="p">()</span> <span class="p">[]</span><span class="kt">string</span>
<span class="nf">Get</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="nx">item</span> <span class="kd">interface</span><span class="p">{},</span> <span class="nx">exists</span> <span class="kt">bool</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">)</span>
<span class="nf">GetByKey</span><span class="p">(</span><span class="nx">key</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="nx">item</span> <span class="kd">interface</span><span class="p">{},</span> <span class="nx">exists</span> <span class="kt">bool</span><span class="p">,</span> <span class="nx">err</span> <span class="kt">error</span><span class="p">)</span>
<span class="nf">Replace</span><span class="p">([]</span><span class="kd">interface</span><span class="p">{},</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">error</span>
<span class="nf">Resync</span><span class="p">()</span> <span class="kt">error</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>Store 接口的方法都比较直观,Store 的实现有很多,我们等下看 Queue 里用到的是哪个实现。</p>
<p>Queue 接口的实现是 FIFO 和 DeltaFIFO 两个类型,我们在 Informer 里用到的是 DeltaFIFO,而 DeltaFIFO 也没有依赖 FIFO,所以下面我们直接看 DeltaFIFO 是怎么实现的。</p>
<h2 id="deltafifo">DeltaFIFO</h2>
<ul>
<li><strong>client-go/tools/cache/delta_fifo.go:97</strong></li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">DeltaFIFO</span> <span class="kd">struct</span> <span class="p">{</span>
<span class="nx">lock</span> <span class="nx">sync</span><span class="p">.</span><span class="nx">RWMutex</span>
<span class="nx">cond</span> <span class="nx">sync</span><span class="p">.</span><span class="nx">Cond</span>
<span class="nx">items</span> <span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">Deltas</span>
<span class="nx">queue</span> <span class="p">[]</span><span class="kt">string</span> <span class="c1">// 这个 queue 里是没有重复元素的,和上面 items 的 key 保持一致
</span><span class="c1"></span> <span class="nx">populated</span> <span class="kt">bool</span>
<span class="nx">initialPopulationCount</span> <span class="kt">int</span>
<span class="nx">keyFunc</span> <span class="nx">KeyFunc</span> <span class="c1">// 用于构造上面 map 用到的 key
</span><span class="c1"></span> <span class="nx">knownObjects</span> <span class="nx">KeyListerGetter</span> <span class="c1">// 用来检索所有的 keys
</span><span class="c1"></span> <span class="nx">closed</span> <span class="kt">bool</span>
<span class="nx">emitDeltaTypeReplaced</span> <span class="kt">bool</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>这里有一个 Deltas 类型,看下具体的定义:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">Deltas</span> <span class="p">[]</span><span class="nx">Delta</span>

<span class="kd">type</span> <span class="nx">Delta</span> <span class="kd">struct</span> <span class="p">{</span>
<span class="nx">Type</span> <span class="nx">DeltaType</span>
<span class="nx">Object</span> <span class="kd">interface</span><span class="p">{}</span>
<span class="p">}</span>

<span class="kd">type</span> <span class="nx">DeltaType</span> <span class="kt">string</span>

<span class="kd">const</span> <span class="p">(</span>
<span class="nx">Added</span> <span class="nx">DeltaType</span> <span class="p">=</span> <span class="s">&#34;Added&#34;</span>
<span class="nx">Updated</span> <span class="nx">DeltaType</span> <span class="p">=</span> <span class="s">&#34;Updated&#34;</span>
<span class="nx">Deleted</span> <span class="nx">DeltaType</span> <span class="p">=</span> <span class="s">&#34;Deleted&#34;</span>
<span class="nx">Replaced</span> <span class="nx">DeltaType</span> <span class="p">=</span> <span class="s">&#34;Replaced&#34;</span>
<span class="nx">Sync</span> <span class="nx">DeltaType</span> <span class="p">=</span> <span class="s">&#34;Sync&#34;</span>
<span class="p">)</span>

</code></pre></td></tr></table>
</div>
</div><p>可以看到 Delta 结构体保存的是 DeltaType(就是一个字符串)和发生了这种 Delta 的具体对象。</p>
<p>DeltaFIFO 内部主要维护的一个队列和一个 map,直观一点表示如下:</p>
<p><img src="https://gitee.com/daniel-hutao/images/raw/master/delta_fifo.png" alt=""></p>
<p>DeltaFIFO 的 New 函数是 <code>NewDeltaFIFOWithOptions()</code></p>
<ul>
<li><strong>client-go/tools/cache/delta_fifo.go:218</strong></li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="nf">NewDeltaFIFOWithOptions</span><span class="p">(</span><span class="nx">opts</span> <span class="nx">DeltaFIFOOptions</span><span class="p">)</span> <span class="o">*</span><span class="nx">DeltaFIFO</span> <span class="p">{</span>
<span class="k">if</span> <span class="nx">opts</span><span class="p">.</span><span class="nx">KeyFunction</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="nx">opts</span><span class="p">.</span><span class="nx">KeyFunction</span> <span class="p">=</span> <span class="nx">MetaNamespaceKeyFunc</span>
<span class="p">}</span>

<span class="nx">f</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="nx">DeltaFIFO</span><span class="p">{</span>
<span class="nx">items</span><span class="p">:</span> <span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">Deltas</span><span class="p">{},</span>
<span class="nx">queue</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{},</span>
<span class="nx">keyFunc</span><span class="p">:</span> <span class="nx">opts</span><span class="p">.</span><span class="nx">KeyFunction</span><span class="p">,</span>
<span class="nx">knownObjects</span><span class="p">:</span> <span class="nx">opts</span><span class="p">.</span><span class="nx">KnownObjects</span><span class="p">,</span>

<span class="nx">emitDeltaTypeReplaced</span><span class="p">:</span> <span class="nx">opts</span><span class="p">.</span><span class="nx">EmitDeltaTypeReplaced</span><span class="p">,</span>
<span class="p">}</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">cond</span><span class="p">.</span><span class="nx">L</span> <span class="p">=</span> <span class="o">&amp;</span><span class="nx">f</span><span class="p">.</span><span class="nx">lock</span>
<span class="k">return</span> <span class="nx">f</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><h2 id="添加元素---queueactionlocked">添加元素 - queueActionLocked()</h2>
<p>可以注意到 DeltaFIFO 的 Add() 等方法等方法体都很简短,大致这样:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="p">(</span><span class="nx">f</span> <span class="o">*</span><span class="nx">DeltaFIFO</span><span class="p">)</span> <span class="nf">Add</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span>
<span class="k">defer</span> <span class="nx">f</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">populated</span> <span class="p">=</span> <span class="kc">true</span>
<span class="k">return</span> <span class="nx">f</span><span class="p">.</span><span class="nf">queueActionLocked</span><span class="p">(</span><span class="nx">Added</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>里面的逻辑就是调用 <code>queueActionLocked()</code> 方法传递对应的 DeltaType 进去,前面提到过 DeltaType 就是 Added、Updated、Deleted 等字符串,所以我们直接先看 <code>queueActionLocked()</code> 方法的实现。</p>
<ul>
<li><strong>client-go/tools/cache/delta_fifo.go:409</strong></li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="p">(</span><span class="nx">f</span> <span class="o">*</span><span class="nx">DeltaFIFO</span><span class="p">)</span> <span class="nf">queueActionLocked</span><span class="p">(</span><span class="nx">actionType</span> <span class="nx">DeltaType</span><span class="p">,</span> <span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
<span class="nx">id</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nf">KeyOf</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span> <span class="c1">// 计算这个对象的 key
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">KeyError</span><span class="p">{</span><span class="nx">obj</span><span class="p">,</span> <span class="nx">err</span><span class="p">}</span>
<span class="p">}</span>
<span class="nx">oldDeltas</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">items</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="c1">// 从 items map 里获取当前的 Deltas
</span><span class="c1"></span> <span class="nx">newDeltas</span> <span class="o">:=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">oldDeltas</span><span class="p">,</span> <span class="nx">Delta</span><span class="p">{</span><span class="nx">actionType</span><span class="p">,</span> <span class="nx">obj</span><span class="p">})</span> <span class="c1">// 构造一个 Delta,添加到 Deltas 中,也就是 []Delta 里
</span><span class="c1"></span> <span class="nx">newDeltas</span> <span class="p">=</span> <span class="nf">dedupDeltas</span><span class="p">(</span><span class="nx">newDeltas</span><span class="p">)</span> <span class="c1">// 如果最近个 Delta 是重复的,则保留后一个;目前版本只处理的 Deleted 重复场景
</span><span class="c1"></span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="nx">newDeltas</span><span class="p">)</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span> <span class="c1">// 理论上 newDeltas 长度一定大于0
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">exists</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">items</span><span class="p">[</span><span class="nx">id</span><span class="p">];</span> <span class="p">!</span><span class="nx">exists</span> <span class="p">{</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">queue</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">f</span><span class="p">.</span><span class="nx">queue</span><span class="p">,</span> <span class="nx">id</span><span class="p">)</span> <span class="c1">// 如果 id 不存在,则在队列里添加
</span><span class="c1"></span> <span class="p">}</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">items</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="p">=</span> <span class="nx">newDeltas</span> <span class="c1">// 如果 id 已经存在,则只更新 items map 里对应这个 key 的 Deltas
</span><span class="c1"></span> <span class="nx">f</span><span class="p">.</span><span class="nx">cond</span><span class="p">.</span><span class="nf">Broadcast</span><span class="p">()</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="c1">// 理论上这里执行不到
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">oldDeltas</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="nx">klog</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; ignoring&#34;</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">oldDeltas</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">nil</span>
<span class="p">}</span>
<span class="nx">klog</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; breaking invariant by storing empty Deltas&#34;</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">oldDeltas</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">items</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="p">=</span> <span class="nx">newDeltas</span>
<span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; broke DeltaFIFO invariant by storing empty Deltas&#34;</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">oldDeltas</span><span class="p">,</span> <span class="nx">obj</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">return</span> <span class="kc">nil</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>到这里再反过来看 Add() Delete() Update() Get() 等函数,就很清晰了,只是将对应变化类型的 obj 添加到队列中。</p>
<h2 id="pop">Pop()</h2>
<p>Pop 按照元素的添加或更新顺序有序返回一个元素(Deltas),在队列为空时会阻塞。另外 Pop 过程会先从队列中删除一个元素然后返回,所以如果处理失败了需要通过 <code>AddIfNotPresent()</code> 方法将这个元素加回到队列中。</p>
<p>Pop 的参数是 <code>type PopProcessFunc func(interface{}) error</code> 类型的 process,中 <code>Pop()</code> 函数中直接将队列里的第一个元素出队,然后丢给 process 处理,如果处理失败会重新入队,但是这个 Deltas 和对应的错误信息会被返回。</p>
<ul>
<li><strong>client-go/tools/cache/delta_fifo.go:515</strong></li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="p">(</span><span class="nx">f</span> <span class="o">*</span><span class="nx">DeltaFIFO</span><span class="p">)</span> <span class="nf">Pop</span><span class="p">(</span><span class="nx">process</span> <span class="nx">PopProcessFunc</span><span class="p">)</span> <span class="p">(</span><span class="kd">interface</span><span class="p">{},</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span>
<span class="k">defer</span> <span class="nx">f</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span>
<span class="k">for</span> <span class="p">{</span> <span class="c1">// 这个循环其实没有意义,和下面的 !ok 一起解决了一个不会发生的问题
</span><span class="c1"></span> <span class="k">for</span> <span class="nb">len</span><span class="p">(</span><span class="nx">f</span><span class="p">.</span><span class="nx">queue</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span> <span class="c1">// 如果为空则进入这个循环
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">f</span><span class="p">.</span><span class="nx">closed</span> <span class="p">{</span> <span class="c1">// 队列关闭则直接返回
</span><span class="c1"></span> <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">ErrFIFOClosed</span>
<span class="p">}</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">cond</span><span class="p">.</span><span class="nf">Wait</span><span class="p">()</span> <span class="c1">// 等待
</span><span class="c1"></span> <span class="p">}</span>
<span class="nx">id</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">queue</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c1">// queue 里放的是 key
</span><span class="c1"></span> <span class="nx">f</span><span class="p">.</span><span class="nx">queue</span> <span class="p">=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">queue</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="c1">// queue 中删除这个 key
</span><span class="c1"></span> <span class="nx">depth</span> <span class="o">:=</span> <span class="nb">len</span><span class="p">(</span><span class="nx">f</span><span class="p">.</span><span class="nx">queue</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">f</span><span class="p">.</span><span class="nx">initialPopulationCount</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="p">{</span> <span class="c1">// 第一次调用 Replace() 插入到元素数量
</span><span class="c1"></span> <span class="nx">f</span><span class="p">.</span><span class="nx">initialPopulationCount</span><span class="o">--</span>
<span class="p">}</span>
<span class="nx">item</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">items</span><span class="p">[</span><span class="nx">id</span><span class="p">]</span> <span class="c1">// 从 items map[string]Deltas 中获取一个 Deltas
</span><span class="c1"></span> <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span> <span class="c1">// 理论上不可能找不到,为此引入了上面的 for 嵌套,感觉不是很好
</span><span class="c1"></span> <span class="nx">klog</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;Inconceivable! %q was in f.queue but not f.items; ignoring.&#34;</span><span class="p">,</span> <span class="nx">id</span><span class="p">)</span>
<span class="k">continue</span>
<span class="p">}</span>
<span class="nb">delete</span><span class="p">(</span><span class="nx">f</span><span class="p">.</span><span class="nx">items</span><span class="p">,</span> <span class="nx">id</span><span class="p">)</span> <span class="c1">// items map 中也删除这个元素
</span><span class="c1"></span> <span class="c1">// 当队列长度超过 10 并且处理一个元素时间超过 0.1 s 时打印日志;队列长度理论上不会变长因为处理一个元素时是阻塞的,这时候新的元素加不进来
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">depth</span> <span class="p">&gt;</span> <span class="mi">10</span> <span class="p">{</span>
<span class="nx">trace</span> <span class="o">:=</span> <span class="nx">utiltrace</span><span class="p">.</span><span class="nf">New</span><span class="p">(</span><span class="s">&#34;DeltaFIFO Pop Process&#34;</span><span class="p">,</span>
<span class="nx">utiltrace</span><span class="p">.</span><span class="nx">Field</span><span class="p">{</span><span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;ID&#34;</span><span class="p">,</span> <span class="nx">Value</span><span class="p">:</span> <span class="nx">id</span><span class="p">},</span>
<span class="nx">utiltrace</span><span class="p">.</span><span class="nx">Field</span><span class="p">{</span><span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;Depth&#34;</span><span class="p">,</span> <span class="nx">Value</span><span class="p">:</span> <span class="nx">depth</span><span class="p">},</span>
<span class="nx">utiltrace</span><span class="p">.</span><span class="nx">Field</span><span class="p">{</span><span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;Reason&#34;</span><span class="p">,</span> <span class="nx">Value</span><span class="p">:</span> <span class="s">&#34;slow event handlers blocking the queue&#34;</span><span class="p">})</span>
<span class="k">defer</span> <span class="nx">trace</span><span class="p">.</span><span class="nf">LogIfLong</span><span class="p">(</span><span class="mi">100</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Millisecond</span><span class="p">)</span>
<span class="p">}</span>
<span class="nx">err</span> <span class="o">:=</span> <span class="nf">process</span><span class="p">(</span><span class="nx">item</span><span class="p">)</span> <span class="c1">// 丢给 PopProcessFunc 处理
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">e</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">err</span><span class="p">.(</span><span class="nx">ErrRequeue</span><span class="p">);</span> <span class="nx">ok</span> <span class="p">{</span> <span class="c1">// 如果需要 requeue 则加回到队列里
</span><span class="c1"></span> <span class="nx">f</span><span class="p">.</span><span class="nf">addIfNotPresent</span><span class="p">(</span><span class="nx">id</span><span class="p">,</span> <span class="nx">item</span><span class="p">)</span>
<span class="nx">err</span> <span class="p">=</span> <span class="nx">e</span><span class="p">.</span><span class="nx">Err</span>
<span class="p">}</span>
<span class="c1">// 返回这个 Deltas 和错误信息
</span><span class="c1"></span> <span class="k">return</span> <span class="nx">item</span><span class="p">,</span> <span class="nx">err</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>我们看一下 Pop() 的实际调用场景:</p>
<ul>
<li><strong>client-go/tools/cache/controller.go:181</strong></li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="p">(</span><span class="nx">c</span> <span class="o">*</span><span class="nx">controller</span><span class="p">)</span> <span class="nf">processLoop</span><span class="p">()</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">{</span>
<span class="nx">obj</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">c</span><span class="p">.</span><span class="nx">config</span><span class="p">.</span><span class="nx">Queue</span><span class="p">.</span><span class="nf">Pop</span><span class="p">(</span><span class="nf">PopProcessFunc</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nx">config</span><span class="p">.</span><span class="nx">Process</span><span class="p">))</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">ErrFIFOClosed</span> <span class="p">{</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="k">if</span> <span class="nx">c</span><span class="p">.</span><span class="nx">config</span><span class="p">.</span><span class="nx">RetryOnError</span> <span class="p">{</span>
<span class="nx">c</span><span class="p">.</span><span class="nx">config</span><span class="p">.</span><span class="nx">Queue</span><span class="p">.</span><span class="nf">AddIfNotPresent</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span> <span class="c1">// 其实 Pop 内部已经调用了 AddIfNotPresent,这里也有点多余;也许更加健壮吧
</span><span class="c1"></span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>到这还有一个疑问,就是 process 函数是怎么实现的?我们看 sharedIndexInformer 里的 process 函数逻辑:</p>
<ul>
<li><strong>client-go/tools/cache/shared_informer.go:537</strong></li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">sharedIndexInformer</span><span class="p">)</span> <span class="nf">HandleDeltas</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="kt">error</span> <span class="p">{</span>
<span class="nx">s</span><span class="p">.</span><span class="nx">blockDeltas</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span>
<span class="k">defer</span> <span class="nx">s</span><span class="p">.</span><span class="nx">blockDeltas</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span>

<span class="c1">// 这个遍历是从旧到新的过程
</span><span class="c1"></span> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">d</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">obj</span><span class="p">.(</span><span class="nx">Deltas</span><span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="nx">d</span><span class="p">.</span><span class="nx">Type</span> <span class="p">{</span>
<span class="k">case</span> <span class="nx">Sync</span><span class="p">,</span> <span class="nx">Replaced</span><span class="p">,</span> <span class="nx">Added</span><span class="p">,</span> <span class="nx">Updated</span><span class="p">:</span> <span class="c1">// 下面一个 case 是 Deleted
</span><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">cacheMutationDetector</span><span class="p">.</span><span class="nf">AddObject</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">old</span><span class="p">,</span> <span class="nx">exists</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">indexer</span><span class="p">.</span><span class="nf">Get</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">);</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="nx">exists</span> <span class="p">{</span>
<span class="c1">// 更新 indexer
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">indexer</span><span class="p">.</span><span class="nf">Update</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">err</span>
<span class="p">}</span>

<span class="nx">isSync</span> <span class="o">:=</span> <span class="kc">false</span>
<span class="k">switch</span> <span class="p">{</span>
<span class="k">case</span> <span class="nx">d</span><span class="p">.</span><span class="nx">Type</span> <span class="o">==</span> <span class="nx">Sync</span><span class="p">:</span>
<span class="nx">isSync</span> <span class="p">=</span> <span class="kc">true</span>
<span class="k">case</span> <span class="nx">d</span><span class="p">.</span><span class="nx">Type</span> <span class="o">==</span> <span class="nx">Replaced</span><span class="p">:</span>
<span class="k">if</span> <span class="nx">accessor</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">meta</span><span class="p">.</span><span class="nf">Accessor</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">);</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">if</span> <span class="nx">oldAccessor</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">meta</span><span class="p">.</span><span class="nf">Accessor</span><span class="p">(</span><span class="nx">old</span><span class="p">);</span> <span class="nx">err</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="nx">isSync</span> <span class="p">=</span> <span class="nx">accessor</span><span class="p">.</span><span class="nf">GetResourceVersion</span><span class="p">()</span> <span class="o">==</span> <span class="nx">oldAccessor</span><span class="p">.</span><span class="nf">GetResourceVersion</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// 更新通知
</span><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">processor</span><span class="p">.</span><span class="nf">distribute</span><span class="p">(</span><span class="nx">updateNotification</span><span class="p">{</span><span class="nx">oldObj</span><span class="p">:</span> <span class="nx">old</span><span class="p">,</span> <span class="nx">newObj</span><span class="p">:</span> <span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">},</span> <span class="nx">isSync</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c1">// 将 obj 加到 indexer 里
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">indexer</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">err</span>
<span class="p">}</span>
<span class="c1">// 添加通知
</span><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">processor</span><span class="p">.</span><span class="nf">distribute</span><span class="p">(</span><span class="nx">addNotification</span><span class="p">{</span><span class="nx">newObj</span><span class="p">:</span> <span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">},</span> <span class="kc">false</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">case</span> <span class="nx">Deleted</span><span class="p">:</span> <span class="c1">// 如果是删除,则从 indexer 中删除 obj
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">indexer</span><span class="p">.</span><span class="nf">Delete</span><span class="p">(</span><span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">err</span>
<span class="p">}</span>
<span class="c1">// 发布一个删除消息
</span><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">processor</span><span class="p">.</span><span class="nf">distribute</span><span class="p">(</span><span class="nx">deleteNotification</span><span class="p">{</span><span class="nx">oldObj</span><span class="p">:</span> <span class="nx">d</span><span class="p">.</span><span class="nx">Object</span><span class="p">},</span> <span class="kc">false</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="kc">nil</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><h2 id="replace">Replace()</h2>
<p>Replace() 简单地做两件事:</p>
<ol>
<li>给传入的对象列表添加一个 Sync/Replace DeltaType 的 Delta</li>
<li>然后执行一些删除逻辑</li>
</ol>
<p>这里的 Replace() 过程可以简单理解成传递一个新的 []Deltas 过来,如果当前 DeltaFIFO 里已经有这些元素,则追加一个 Sync/Replace 动作,反之 DeltaFIFO 里多出来的 Deltas 则可能是与 apiserver 失联导致实际已经删除,但是删除动作没有 watch 到的那些对象,所以直接追加一个 Deleted 的 Delta;</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span><span class="lnt">12
</span><span class="lnt">13
</span><span class="lnt">14
</span><span class="lnt">15
</span><span class="lnt">16
</span><span class="lnt">17
</span><span class="lnt">18
</span><span class="lnt">19
</span><span class="lnt">20
</span><span class="lnt">21
</span><span class="lnt">22
</span><span class="lnt">23
</span><span class="lnt">24
</span><span class="lnt">25
</span><span class="lnt">26
</span><span class="lnt">27
</span><span class="lnt">28
</span><span class="lnt">29
</span><span class="lnt">30
</span><span class="lnt">31
</span><span class="lnt">32
</span><span class="lnt">33
</span><span class="lnt">34
</span><span class="lnt">35
</span><span class="lnt">36
</span><span class="lnt">37
</span><span class="lnt">38
</span><span class="lnt">39
</span><span class="lnt">40
</span><span class="lnt">41
</span><span class="lnt">42
</span><span class="lnt">43
</span><span class="lnt">44
</span><span class="lnt">45
</span><span class="lnt">46
</span><span class="lnt">47
</span><span class="lnt">48
</span><span class="lnt">49
</span><span class="lnt">50
</span><span class="lnt">51
</span><span class="lnt">52
</span><span class="lnt">53
</span><span class="lnt">54
</span><span class="lnt">55
</span><span class="lnt">56
</span><span class="lnt">57
</span><span class="lnt">58
</span><span class="lnt">59
</span><span class="lnt">60
</span><span class="lnt">61
</span><span class="lnt">62
</span><span class="lnt">63
</span><span class="lnt">64
</span><span class="lnt">65
</span><span class="lnt">66
</span><span class="lnt">67
</span><span class="lnt">68
</span><span class="lnt">69
</span><span class="lnt">70
</span><span class="lnt">71
</span><span class="lnt">72
</span><span class="lnt">73
</span><span class="lnt">74
</span><span class="lnt">75
</span><span class="lnt">76
</span><span class="lnt">77
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="p">(</span><span class="nx">f</span> <span class="o">*</span><span class="nx">DeltaFIFO</span><span class="p">)</span> <span class="nf">Replace</span><span class="p">(</span><span class="nx">list</span> <span class="p">[]</span><span class="kd">interface</span><span class="p">{},</span> <span class="nx">_</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nf">Lock</span><span class="p">()</span>
<span class="k">defer</span> <span class="nx">f</span><span class="p">.</span><span class="nx">lock</span><span class="p">.</span><span class="nf">Unlock</span><span class="p">()</span>
<span class="nx">keys</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="nx">sets</span><span class="p">.</span><span class="nx">String</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="nx">list</span><span class="p">))</span> <span class="c1">// 用来保存 list 中每个 item 的 key
</span><span class="c1"></span>
<span class="c1">// 老代码兼容逻辑
</span><span class="c1"></span> <span class="nx">action</span> <span class="o">:=</span> <span class="nx">Sync</span>
<span class="k">if</span> <span class="nx">f</span><span class="p">.</span><span class="nx">emitDeltaTypeReplaced</span> <span class="p">{</span>
<span class="nx">action</span> <span class="p">=</span> <span class="nx">Replaced</span>
<span class="p">}</span>

<span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">item</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">list</span> <span class="p">{</span> <span class="c1">// 在每个 item 后面添加一个 Sync/Replaced 动作
</span><span class="c1"></span> <span class="nx">key</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nf">KeyOf</span><span class="p">(</span><span class="nx">item</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">KeyError</span><span class="p">{</span><span class="nx">item</span><span class="p">,</span> <span class="nx">err</span><span class="p">}</span>
<span class="p">}</span>
<span class="nx">keys</span><span class="p">.</span><span class="nf">Insert</span><span class="p">(</span><span class="nx">key</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nf">queueActionLocked</span><span class="p">(</span><span class="nx">action</span><span class="p">,</span> <span class="nx">item</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;couldn&#39;t enqueue object: %v&#34;</span><span class="p">,</span> <span class="nx">err</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>

<span class="k">if</span> <span class="nx">f</span><span class="p">.</span><span class="nx">knownObjects</span> <span class="o">==</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="nx">queuedDeletions</span> <span class="o">:=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="nx">k</span><span class="p">,</span> <span class="nx">oldItem</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">f</span><span class="p">.</span><span class="nx">items</span> <span class="p">{</span> <span class="c1">// 删除 f.items 里的老元素
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">keys</span><span class="p">.</span><span class="nf">Has</span><span class="p">(</span><span class="nx">k</span><span class="p">)</span> <span class="p">{</span>
<span class="k">continue</span>
<span class="p">}</span>

<span class="kd">var</span> <span class="nx">deletedObj</span> <span class="kd">interface</span><span class="p">{}</span>
<span class="k">if</span> <span class="nx">n</span> <span class="o">:=</span> <span class="nx">oldItem</span><span class="p">.</span><span class="nf">Newest</span><span class="p">();</span> <span class="nx">n</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> <span class="c1">// 如果 Deltas 不为空则有返回值
</span><span class="c1"></span> <span class="nx">deletedObj</span> <span class="p">=</span> <span class="nx">n</span><span class="p">.</span><span class="nx">Object</span>
<span class="p">}</span>
<span class="nx">queuedDeletions</span><span class="o">++</span>
<span class="c1">// 标记删除;因为和 apiserver 失联引起的删除状态没有及时获取到,所以这里是 DeletedFinalStateUnknown 类型
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nf">queueActionLocked</span><span class="p">(</span><span class="nx">Deleted</span><span class="p">,</span> <span class="nx">DeletedFinalStateUnknown</span><span class="p">{</span><span class="nx">k</span><span class="p">,</span> <span class="nx">deletedObj</span><span class="p">});</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">err</span>
<span class="p">}</span>
<span class="p">}</span>

<span class="k">if</span> <span class="p">!</span><span class="nx">f</span><span class="p">.</span><span class="nx">populated</span> <span class="p">{</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">populated</span> <span class="p">=</span> <span class="kc">true</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">initialPopulationCount</span> <span class="p">=</span> <span class="nx">keys</span><span class="p">.</span><span class="nf">Len</span><span class="p">()</span> <span class="o">+</span> <span class="nx">queuedDeletions</span>
<span class="p">}</span>

<span class="k">return</span> <span class="kc">nil</span>
<span class="p">}</span>

<span class="nx">knownKeys</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">knownObjects</span><span class="p">.</span><span class="nf">ListKeys</span><span class="p">()</span> <span class="c1">// key 就是例如 &#34;default/pod_1&#34; 这种字符串
</span><span class="c1"></span> <span class="nx">queuedDeletions</span> <span class="o">:=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">k</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">knownKeys</span> <span class="p">{</span>
<span class="k">if</span> <span class="nx">keys</span><span class="p">.</span><span class="nf">Has</span><span class="p">(</span><span class="nx">k</span><span class="p">)</span> <span class="p">{</span>
<span class="k">continue</span>
<span class="p">}</span>
<span class="c1">// 新列表里不存在的老元素标记为将要删除
</span><span class="c1"></span> <span class="nx">deletedObj</span><span class="p">,</span> <span class="nx">exists</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">knownObjects</span><span class="p">.</span><span class="nf">GetByKey</span><span class="p">(</span><span class="nx">k</span><span class="p">)</span>
<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="nx">deletedObj</span> <span class="p">=</span> <span class="kc">nil</span>
<span class="nx">klog</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;Unexpected error %v during lookup of key %v, placing DeleteFinalStateUnknown marker without object&#34;</span><span class="p">,</span> <span class="nx">err</span><span class="p">,</span> <span class="nx">k</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">!</span><span class="nx">exists</span> <span class="p">{</span>
<span class="nx">deletedObj</span> <span class="p">=</span> <span class="kc">nil</span>
<span class="nx">klog</span><span class="p">.</span><span class="nf">Infof</span><span class="p">(</span><span class="s">&#34;Key %v does not exist in known objects store, placing DeleteFinalStateUnknown marker without object&#34;</span><span class="p">,</span> <span class="nx">k</span><span class="p">)</span>
<span class="p">}</span>
<span class="nx">queuedDeletions</span><span class="o">++</span>
<span class="c1">// 添加一个删除动作;因为与 apiserver 失联等场景会引起删除事件没有 wathch 到,所以是 DeletedFinalStateUnknown 类型
</span><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nf">queueActionLocked</span><span class="p">(</span><span class="nx">Deleted</span><span class="p">,</span> <span class="nx">DeletedFinalStateUnknown</span><span class="p">{</span><span class="nx">k</span><span class="p">,</span> <span class="nx">deletedObj</span><span class="p">});</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">err</span>
<span class="p">}</span>
<span class="p">}</span>

<span class="k">if</span> <span class="p">!</span><span class="nx">f</span><span class="p">.</span><span class="nx">populated</span> <span class="p">{</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">populated</span> <span class="p">=</span> <span class="kc">true</span>
<span class="nx">f</span><span class="p">.</span><span class="nx">initialPopulationCount</span> <span class="p">=</span> <span class="nx">keys</span><span class="p">.</span><span class="nf">Len</span><span class="p">()</span> <span class="o">+</span> <span class="nx">queuedDeletions</span>
<span class="p">}</span>

<span class="k">return</span> <span class="kc">nil</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>这里有一个 knownObjects 属性,要完整理解 Replace() 逻辑还得看下 knownObjects 是什么逻辑。</p>
<p>我们去跟 knownObjects 属性的初始化,可以看到其引用的是 cache 类型实现的 Store,cache 是实现 Indexer 的那个 cache,Indexer 的源码分析可以在我的另外一篇文章<a href="../client-go-indexer/">《Kubernetes client-go Indexer / ThreadSafeStore 源码分析》</a> 中看到。</p>
<ul>
<li><strong>client-go/tools/cache/store.go:258</strong></li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="kd">func</span> <span class="nf">NewStore</span><span class="p">(</span><span class="nx">keyFunc</span> <span class="nx">KeyFunc</span><span class="p">)</span> <span class="nx">Store</span> <span class="p">{</span>
<span class="k">return</span> <span class="o">&amp;</span><span class="nx">cache</span><span class="p">{</span>
<span class="nx">cacheStorage</span><span class="p">:</span> <span class="nf">NewThreadSafeStore</span><span class="p">(</span><span class="nx">Indexers</span><span class="p">{},</span> <span class="nx">Indices</span><span class="p">{}),</span>
<span class="nx">keyFunc</span><span class="p">:</span> <span class="nx">keyFunc</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>这里是当作一个 Store 来用,而不是 Indexer。中 NewStore() 函数调用时传递的参数是:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="nx">clientState</span> <span class="o">:=</span> <span class="nf">NewStore</span><span class="p">(</span><span class="nx">DeletionHandlingMetaNamespaceKeyFunc</span><span class="p">)</span>
</code></pre></td></tr></table>
</div>
</div><div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span><span class="lnt">4
</span><span class="lnt">5
</span><span class="lnt">6
</span><span class="lnt">7
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="c1">// 处理了 DeletedFinalStateUnknown 对象获取 key 问题
</span><span class="c1"></span><span class="kd">func</span> <span class="nf">DeletionHandlingMetaNamespaceKeyFunc</span><span class="p">(</span><span class="nx">obj</span> <span class="kd">interface</span><span class="p">{})</span> <span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="nx">d</span><span class="p">,</span> <span class="nx">ok</span> <span class="o">:=</span> <span class="nx">obj</span><span class="p">.(</span><span class="nx">DeletedFinalStateUnknown</span><span class="p">);</span> <span class="nx">ok</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">d</span><span class="p">.</span><span class="nx">Key</span><span class="p">,</span> <span class="kc">nil</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nf">MetaNamespaceKeyFunc</span><span class="p">(</span><span class="nx">obj</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></td></tr></table>
</div>
</div><p>所以 knownObjects 通过 cache 类型实例,使用了和 Indexer 类似的机制,通过内部 ThreadSafeStore 来实现了检索队列所有元素的 keys 的能力。</p>
<p>DeltaFIFO 和 Indexer 之间还有一个桥梁 Informer,我们这里简单提到了 <em>sharedIndexInformer</em> 的 <em>HandleDeltas()</em> 方法,后面详细分析 Informer 的逻辑,最终再将整个自定义控制器和 client-go 相关组件逻辑串在一起。</p>
<p>(转载请保留本文原始链接)</p>

</div>

<div class="post-copyright">
<p class="copyright-item">
<span class="item-title">Author</span>
<span class="item-content">Daniel Hu</span>
</p>
<p class="copyright-item">
<span class="item-title">LastMod</span>
<span class="item-content">
2021-09-29
<a href="/commit/d4a62f6eb6eb8899ca7e1fe814136fc44a1ba64a" title="add client-go-indexer">(d4a62f6)</a>
</span>
</p>


</div>
<footer class="post-footer">
<div class="post-tags">
<a href="/tags/k8s/">k8s</a>
<a href="/tags/client-go/">client-go</a>
<a href="/tags/deltafifo/">deltafifo</a>
<a href="/tags/%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90/">源码分析</a>
</div>
<nav class="post-nav">

<a class="next" href="/post/k8s/client-go-indexer/">
<span class="next-text nav-default">Kubernetes client-go Indexer / ThreadSafeStore 源码分析</span>
<span class="next-text nav-mobile">Next</span>
<i class="iconfont icon-right"></i>
</a>
</nav>
</footer>
</article>
</div>

 


<script src="https://utteranc.es/client.js"
repo="daniel-hutao/hugoblogtalks"
issue-term="pathname"
theme="github-light"
crossorigin="anonymous"
async>
</script>
<noscript>Please enable JavaScript to view the <a href="https://github.com/utterance">comments powered by utterances.</a></noscript>

</div>
</main>

<footer id="footer" class="footer">
<div class="social-links">
<a href="mailto:farmer.hutao@outlook.com" class="iconfont icon-email" title="email"></a>
<a href="https://github.com/daniel-hutao" class="iconfont icon-github" title="github"></a>
<a href="https://danielhu.cn/index.xml" type="application/rss+xml" class="iconfont icon-rss" title="rss"></a>
</div>

<div class="copyright">
<span class="power-by">
Powered by <a class="hexo-link" href="https://gohugo.io">Hugo</a>
</span>
<span class="division">|</span>
<span class="theme-info">
Theme -
<a class="theme-link" href="https://github.com/olOwOlo/hugo-theme-even">Even</a>
</span>

 

<span class="copyright-year">
&copy;
2021<span class="heart"><i class="iconfont icon-heart"></i></span><span>Daniel Hu</span>
</span>
</div>

</footer>

<div class="back-to-top" id="back-to-top">
<i class="iconfont icon-up"></i>
</div>
</div>

<script src="https://cdn.jsdelivr.net/npm/jquery@3.2.1/dist/jquery.min.js" integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4=" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/slideout@1.0.1/dist/slideout.min.js" integrity="sha256-t+zJ/g8/KXIJMjSVQdnibt4dlaDxc9zXr/9oNPeWqdg=" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@fancyapps/fancybox@3.1.20/dist/jquery.fancybox.min.js" integrity="sha256-XVLffZaxoWfGUEbdzuLi7pwaUJv1cecsQJQqGLe7axY=" crossorigin="anonymous"></script>

 

<script type="text/javascript" src="/js/main.min.c99b103c33d1539acf3025e1913697534542c4a5aa5af0ccc20475ed2863603b.js"></script>


<script type="application/javascript">
var doNotTrack = false;
if (!doNotTrack) {
window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date;
ga('create', 'G-994J9VCT2Q', 'auto');
ga('set', 'anonymizeIp', true);
ga('send', 'pageview');
}
</script>
<script async src='https://www.google-analytics.com/analytics.js'></script>

 

 

 

</body>
</html>

 

随时随地学软件编程-关注百度小程序和微信小程序
关于找一找教程网

本站文章仅代表作者观点,不代表本站立场,所有文章非营利性免费分享。
本站提供了软件编程、网站开发技术、服务器运维、人工智能等等IT技术文章,希望广大程序员努力学习,让我们用科技改变世界。
[Kubernetes client-go DeltaFIFO 源码分析]http://www.zyiz.net/tech/detail-235645.html

上一篇:Service Mesh服务网格技术探究---VMWare+k8s集群+Istio系列:Istio示例应用Bookinfo的部署

下一篇:K8ssandra入门-详细记录在Linux上部署K8ssandra到Kubernetes

赞(0)
关注微信小程序
程序员编程王-随时随地学编程

扫描二维码或查找【程序员编程王】

可以随时随地学编程啦!

技术文章导航 更多>
扫一扫关注最新编程教程