K8s 1.12 Release Information

Announcements
,
1

The kubernetes 1.12 release cycle is underway!

Detailed Schedule: http://bit.ly/k8s112-release-info

We have our set of volunteers for the release team listed here: https://git.k8s.io/sig-release/releases/release-1.12/release_team.md

The anticipated release schedule is posted (https://git.k8s.io/sig-release/releases/release-1.12/release-1.12.md) and key dates are also noted on the community calendar and below.

The primary process point to note for the cycle is that we aim to continue as in the prior release with a shorted code slush and code freeze period. This change seems to have gone well. But it is only possible with continued emphasis on improving test coverage and reliability and keeping a clean CI signal thoroughly the cycle especially for release-master-blocking and release-master-upgrade. The release team may be forced to pull in code freeze in order to give more stabilization time if tests are neglected, but will work throughout the cycle to avoid this and communicate more vigorously early in August if things are not trending positively.

As you may have seen already on k/dev, our features lead is actively collating feature information from SIGs. In addition to collecting information on planned feature coding, we want to remind SIGs of the need to have test cases demonstrating code readiness and early indication of whether new test cases are intended as release blocking as we head toward feature freeze at the end of July.

Similarly, we’re asking feature developers to make an earlier effort in this cycle to draft initial rough documentation and begin coordination with the docs team, as we’re finding documentation has trailed and this leads to lots of extra effort right at the end of the cycle.

Key Dates

Feature freeze: July 31, 2018

Begin code slush: Aug. 28, 2018

Begin code freeze: Sept. 4, 2018

End code freeze: Sept. 19, 2018

Release date: Sept. 25, 2018

On behalf of the 1.12 release team, I want to express our excitement. This is an amazing, collaborative community and we look forward to seeing what the release brings!

1 Like
2

As the month comes to a close our 1.12 release feature freeze is just around the corner (EOD Tuesday July 31).

For anybody unfamiliar with the release process cycle, this is just a feature definition milestone. Code freeze is yet to come, leaving many weeks yet for feature implementation.

The release team and its features lead Stephen Augustus have been checking in with SIGs on feature plans for the release and working to insure the features repo (https://git.k8s.io/features) is up to date. Features not captured and tracked ahead of the feature freeze must go through the exception process (https://git.k8s.io/features/EXCEPTIONS.md).

The current feature list is tracked at http://bit.ly/k8s112-features. If any of your SIG features are missing, please reach out to us ASAP.

SIGs should also be thinking about their release themes (major work focuses) for the 1.12 release notes in addition to insuring those themes are represented in feature issues as described above and have plans for documentation and test coverage. These release themes are a good opportunity for SIGs to communicate with users and contributors on focus areas for the coming weeks, and will feature in an upcoming Blog - Kubernetes posting.

Key Dates

Feature freeze: July 31, 2018

Begin code slush: Aug. 28, 2018

Begin code freeze: Sept. 4, 2018

End code freeze: Sept. 19, 2018

Release date: Sept. 25, 2018

3

As of Tuesday July 31 we’ve passed our feature definition milestone. Top level information on the set of features captured is viewable at:

and:

For more human readable information, watch for a blog post on http://kubernetes.io/blog in the coming weeks with more discussion on pending features.

Again this is feature definition only, there are many weeks yet for implementation ahead of code freeze.

If your SIG feature did not make the deadline yesterday, there is an exception process (https://git.k8s.io/features/EXCEPTIONS.md ).

We did not hit our yesterday target for cutting an alpha release for 1.12, but have had an exciting day today working through updating the documentation on and exercising of the build and release mechanism. Googlers have been working away lately shifting the build and release mechanics to be ones a non-Google employee can finally run. To my knowledge prior to 1.11.1 and 1.12.0-alpha.1 all kubernetes releases have been cut by a Googler, so this is a major and significant shift toward the community. The 1.11.1 release had a hiccup and it appears we’ve proved that fixed for the release of 1.12.0-alpha.1 today. We did have a few other hiccups on 1.12.0-alpha.1 though, but all things considered those look minor and fixable and artifacts are now live:

As you may have seen the alpha release notification email got slightly wedged in the ether and then our workaround had an issue too. Better the artifacts ship first and the announcement late, than pre-announcing incomplete artifacts! And if a missing Content-Type: text/html is the worst of our issues, we’re in great shape considering all that has changed!

Big thanks to Doug MacEachern for sticking his neck out as 1.12 release branch manager during this transition and the many Googlers (Caleb Miles, Ben Elder, and many more behind the scenes) for doing the intrepid pathfinding and debugging to improve this aspect of the release process!!!

Our next major milestone will be the shift into code freeze as September arrives. While this is weeks away, this time always goes by FAST! We request you continue to give consideration to documentation and test cases for your features as you’re developing, as well as keep an eye on CI results related to your SIG and be responsive to requests for issue and test failure triage by the release team. It is imperative that we continually improve our CI signal, maintain passing test status, and ultimately achieve a quality release!

Key Dates

Begin code slush: Aug. 28, 2018

Begin code freeze: Sept. 4, 2018

End code freeze: Sept. 19, 2018

Release date: Sept. 25, 2018

Detailed schedule available at http://bit.ly/k8s112-release-info

1 Like
5

One month to go!!!

Key dates reminder:

  • Feature docs PRs Aug. 21 (past due!)
  • Code Slush Aug. 28
  • Code Freeze Sept. 4
  • Feature docs due Sept. 7
  • Release Target Sept. 25

At an absolute minimum, all tracked features should now have open a placeholder WIP pull request indicating what documentation you will be updating before release. All feature owners have had 1:1 communications on this over the past while. If your PR is not open, you’re past due…please help the docs team help you.

SIG leadership please start limiting code merges in line with the spirit of code slush.

All features are expected to be code complete, including test and open docs PRs, by the start of code freeze on Tuesday, September 4th, 2018.

Overall CI Signal ahead of code freeze is troubling with a dozen failure scenarios currently across sig-release-1.12-all, sig-release-1.12-blocking, sig-release-master-blocking, and sig-release-master-upgrade. A part of active SIG leadership roles is watching and directing your teams toward resolving CI issues, but release team requests for triage help have met with relatively slow responses and the list of failing tests is growing not shrinking. I recognize it’s a time of year where many in the northern hemisphere are in and out on vacations, plus GitHub and Slack are noisy channels and folks are aggressively working toward feature completion. But our ability to efficiently spot problems as larger features merge is dependent on a generally clean CI signal, so anything the broader community can do here to step up test failure analysis would be appreciated.

If there isn’t significant CI signal improvement over the next two weeks the risk of code-freeze elongating and release delay will be high.

1 Like
6

The freeze is here. We’ve entered 1.12 code freeze as of 5pm Pacific today Sept. 4, 2018. And beta.1 announce is probably racing this email through the ether.

A small heads up on freeze: today the code freeze is implemented in the submit queue, but we hope to shift to the new and shiny tide implementation in the next week. A change in user experience might be perceived through this transition, but we expectation that user/GitHub/Tide interactions will be more consistent and intuitive going forward and you’ll be happier for it.

Release target remains Sept. 25 , which is three weeks away. If we have stabilization, master branch will re-open for 1.13 development in just two weeks’ time.

Right now that looks like a large “if”. While a distracting amount of test issues have been GCE/GKE specific cases in the past weeks, and even given Google’s awesome commitment to the CNCF announced last week to better enable community driven testing on community managed test infrastructure resources, we as a community have a long way to go over the coming months and releases to broaden test automation and get better A/B quality signal across use cases across cloud providers. Within this we are today encountering and needing fixing on what do end up being core issues, so we do continue to need SIG attention triaging daily new test failures across sig-release-1.12-all, sig-release-1.12-blocking, sig-release-master-blocking, and sig-release-master-upgrade, even when the test buckets have a ‘G’ in front of their name.

Similarly the release team needs SIG leadership to diligently be marking in GitHub both issues and PRs with SIG, kind, priority and milestone. And regularly update status. Where help is needed we can rally it, but we need visibility.

Documentation PRs are expected in a merge ready state by Sept. 18 at the LATEST.

Release notes will see increased activity this week and next, and hopefully folks find this release’s trial of hack.md more efficient than past workflows.

More links and timeline details at https://github.com/kubernetes/sig-release/blob/master/releases/release-1.12/README.md

2 Likes
7

We are ten (10!!!) days from scheduled release and the release is at risk.

Key upcoming dates:

  • Monday Sept. 17: announce release delay?
  • Wednesday Sept. 19: scheduled end of code freeze, unlikely?
  • Friday Sept. 21: scheduled cherry pick deadline
  • Tuesday Sept. 25: scheduled release target

We’re making notable progress this week on addressing a large variety of test failures on both master and release-1.12 branches.

At this point though the release is at risk, with a decision point coming on Monday regarding delaying release a few days. We’ve had some especially complicated scalability test failures, and it can take many days to validate expected fixes. By Monday we expect a fresh set of data to inform decisions.

There’s also a set of other issues still. But shout outs to Michelle Au, Solly Ross, Ben Elder, Zihong Zheng, Wojchiech Tyczynski, Shyam JVS, Bharanidharan Seetharaman, and Sebastien Vas for their recent detective work (as well as everybody who’s submitted critical-urgent bugfixes this week). Most issues are looking at least understood, if not yet fully resolved.

SIG Leaders:

  • If you haven’t provided input on release major themes yet, please scan your inbox for email subject “Major Themes Needed For 1.12 Release” from Dave Strebel on Sept. 11 requesting your information by…today!
  • And if you’ve not yet looked through the release notes draft in general plus specifically reviewed your SIG’s section, please do so.

Finally a thank you to all feature owners who’ve submitted feature documentation in the past weeks. As of this week all tracked features have their docs in!

8

(From Sept. 17, 2018)

A quick note as we start the penultimate week of the 1.12 release cycle:

Our anticipated test CI improvements over the weekend did not fully materialize, with a new scalability issue even having been raised. Given this, there’s no reason to expect we achieve the master branch code thaw target of Sept. 19. After discussion with our test-infra folks and the broader 1.12 release team, we’re thinking the soonest the master branch will thaw may be Monday Sept. 24 .

The release target continues to look like it needs to slip a couple of days. Final call will be made on that later today.

9

(Also from Sept. 17, 2018)

We’re still working to get a set of fixes in and test proved in Scalability, HPA, and Storage. Officially delaying release day to Thursday Sept. 27.

Master remains frozen as we use master and release-1.12 branch test cases to bolster confidence in fixes. Only ABSOLUTELY critical urgent bug fixes should be being approved for merge. Our hope is to have master open for 1.13 dev early in the week of Sept. 24.

Pending CI Signal, we intend to cut an RC tomorrow.

10

We’re down to a week left until the (modified) release date and a few dates have shifted along with the overall release target.

Key upcoming events barring any new test issues:

  • Friday Sept 21 “cherry-pick deadline” (non-event given master hasn’t yet thawed)
  • Friday Sept 21 cut RC2 and built rpms/debs
  • Monday Sept 24 last release-1.12 branch fast-forward from master branch
  • Monday Sept 24 thaw master branch
  • …final soak: cherry picks only for absolutely critical show stopper bugs…
  • Thursday Sept 27 release

This week started with the team very focused on a few last issues around scalability, storage, and the horizontal pod autoscaler. We’ve deliberately merged PRs staged across time to give more distinct CI signal, and have been fighting against a few hosting issues in the test infrastructure. Things look notably better. But…

We’ve found an issue around RPM/Deb building, have a few upgrade issues to resolve, an issue with Docker 18.03 vs 18.06, and the unfortunately late discovery that we mistakenly did not move the build to go 1.10.4 back in August when that was released. This is a pretty short list of nevertheless significant issues. With everybody’s hard work, and a little luck, I’m hopeful we’ll get this all sorted by the Friday and have a clear green CI signal across all tests early next week.

11

As you might have noticed code thaw’s happened / happening: https://prow.k8s.io/tide

Master is now open for 1.13 development. Only the absolute most critically urgent bug fixes might be cherry picked back in time for 1.12.0, otherwise we’ll be transitioning to the normal cherry pick process for 1.12.y post-release stable patch management.

The release is on target for Thursday, pending final resolution of one node scheduling issue around node taints and un-schedulable nodes across upgrades.

We’re targeting our release retrospective for the Community Meeting October 4. Please add any comments you’d like included in discussion for things that worked well and things that should change in our 1.12 retrospective document.

1 Like
12

Kubernetes official release v1.12.0

https://github.com/kubernetes/kubernetes/releases/tag/v1.12.0

Features

Some highlights include:

  • Stable:
    • Kubelet TLS Bootstrap
    • Azure Virtual Machine Scale Sets and Cluster-Autoscaler
  • Beta:
    • Topology aware dynamic provisioning of storage
    • Configurable pod process namespace sharing
    • Taint node by condition
  • Alpha:
    • RuntimeClass exposing container runtime properties
    • Snapshot / restore of PV’s with CSI volume drivers

Other Information

1 Like