37 stories
·
0 followers

Fleet Below Deck, Part III — State Management

1 Comment
BackstageNews
Vitaly Bragilevsky

Read this post in other languages:
Français, 한국어, 简体中文

This is a multipart series on building Fleet, a next-generation IDE by JetBrains.

In previous parts of this series, we looked at an overview of the Fleet architecture and discussed the algorithms and data structures that are used under the hood in the editor. In this part, we’ll start looking at the approach we take to implement state management. This is a complicated topic, so we’ll devote a couple of blog posts to it. For now, we’ll focus on how we represent and store elements of the application state. In the next part, we’ll talk more about transactional mechanisms around state management in Fleet.

Fleet has a lot of moving parts and performs many different operations, including:

  • Rendering UI elements and interacting with users.
  • Interacting with other services to obtain data and update UI elements.
  • Dealing with files such as saving, loading, parsing, and displaying the differences between them.
  • Orchestrating back-ends that deal with code insight, completion, and search results.

Many of these are complex operations that can degrade the responsiveness of the interface. Fleet is a distributed application, so it may have several frontends distributed over the network – this complicates things even further. Nevertheless, we have to display all of the information for our users consistently and correctly and guarantee they can work harmoniously between their frontends.

In terms of state management, all of these operations boil down to either reading or updating states. UI elements read states to provide users with the actual data, while users update states by editing their documents and moving things around. There are thousands of such operations taking place every minute. All of this makes proper state management a key element of Fleet.

Our principles

JetBrains has been developing IDEs for more than 20 years. Our experience has led us to the following guiding principles regarding state management in Fleet:

Principle 1: Don’t block anyone

Living in a concurrent world is hard. In Kotlin (and in Fleet) we use lightweight concurrency primitives, called coroutines, to organize our concurrent code. While reading states from many coroutines at the same time creates almost no problems, mutating them can be dangerous. The traditional approach is to acquire a lock for a single writer thread, which causes a long waiting queue to read something. We believe this is inappropriate – it should be acceptable for readers to read a potentially slightly outdated state without any delays. To achieve this behavior, we use a variation on the MVCC (multiversion concurrency control) model to access state elements from coroutines. These coroutines either read some version of the state or mutate the state by providing the new version of it. We read and mutate the state in transactions which are much easier to implement under MVCC.

Principle 2: Be efficiently reactive

States change all the time and the UI should reflect these changes as quickly as possible. If you’ve ever programmed simple animations with your first programming language, you know how to do that: erase everything and redraw it from scratch. Unfortunately, full redrawing takes a lot of time. A better idea is to redraw the part that was changed. To do that, we need a way to determine what exactly has changed. The fewer changes, the better. Once we’ve located the changed part of the state, we need to decide as quickly as possible what depends on that part and execute the corresponding coroutine. We have to be efficient in our reactions to state changes.

Principle 3: Represent data wisely

The first two principles are no more than good declarations without the third one. We have to think hard about the way we store and process our data. Storage with highly efficient lookup and mutate operations is no longer an area exclusive to database system implementers. Fleet, being a distributed IDE, requires all of this too. To fulfill our needs, we had to develop our own internal database solution that would be both flexible and performant enough.

What is a state?

There are three ideas we need to consider when thinking about states in Fleet.

Firstly, it’s represented as a persistent data structure with different versions that model change in time. One way to describe such a world is a linear sequence of epochs that go one after another, known as an epochal time model. All interested parties (yes, coroutines!) always read one of the epochs, but not necessarily the most recent one.

Secondly, our state is a database of entities that contain information about everything you see on your screen and everything we hide under the hood. As with many databases, these entities relate to each other in various ways. 

Thirdly, the state and its mutations boil down to basic triples, called datoms, which are primitive data items that allow us to achieve the efficiency we need. Let’s discuss these ideas in a little more detail.

An epochal time model

For a long time, our programs mutated state. Unfortunately, it’s almost never enough to update just one variable. Usually, we have to change many of them one after another in a consistent way. What if someone observes our state in a half-baked form or even attempts mutating it? Imagine that we’ve increased the string’s length but haven’t provided new content. Our users definitely shouldn’t be able to see that. The idea is to hide inconsistent states behind some façade. Coming from one consistent state to the next takes time. It’s like how an epoch in time follows another.

The epochal time model was first explained to the wider programming community by Rich Hickey in his beautiful talk Are We There Yet (see the transcript), devoted to his ideas on implementing the Clojure programming language. What he talks about is that for some time our programs can live in an immutable, consistent world. Immutability makes many things easier to implement, but it’s impossible to stay in the same world forever. As a result of state writers’ activities, a new immutable, consistent world always follows the previous one.

Fleet’s state is accessible in the form of an immutable snapshot, a collection of all the state elements with guaranteed consistency between them. In this model, updating the state creates a new snapshot. To guarantee consistency as states change, we implement transactions.

Fleet has a component called the Kernel, which is responsible for transitioning snapshots as a consequence of state writers’ activities and providing a reference to the most recent snapshot. Interested parties, both readers and writers, may obtain this reference when they need it, but they can’t be sure that this reference corresponds to the most recent version of the world by the time they use it. The Kernel is also responsible for broadcasting changes to the parties that depend on them. The nice thing is that we don’t need to subscribe manually – it’s enough to read some value to then be notified about its changes in the future.

Writers line up to create new snapshots, but readers are never blocked. However, they can receive slightly outdated information.

The data model for our state

Now we are ready to answer the question, what’s in our state? Well, literally everything: document content with its corresponding file information, all the inferred information about that content, caret positions, plugins loaded and their configuration, views and panels locations, etc. The corresponding data model is described in Fleet via Kotlin interfaces as:

interface DocumentFileEntity : SharedEntity {

var document: DocumentEntity

var fileAddress: FileAddress

var readCharset: DetectedCharset

interface DocumentEntity : SharedEntity {

interface DocumentFileEntity : SharedEntity { @Unique @CascadeDeleteBy var document: DocumentEntity @Unique var fileAddress: FileAddress var readCharset: DetectedCharset // ... } interface DocumentEntity : SharedEntity { var text: Text var writable: Boolean // ... }

interface DocumentFileEntity : SharedEntity {
 @Unique
 @CascadeDeleteBy
 var document: DocumentEntity

 @Unique
 var fileAddress: FileAddress

 var readCharset: DetectedCharset
 // ...
}

interface DocumentEntity : SharedEntity {
 var text: Text
 var writable: Boolean
 // ...
}

Note: The type Text is in fact a rope which we covered in the previous part of this series.

We use property annotations to describe entity components and the relationships between them. In this example, a document file entity describes the relation between a unique file on a disk drive and a unique document we’ve read from it. The corresponding document entity should be deleted when the document file entity is deleted.

To maintain such a database of entities, we’ve implemented our own database engine, called RhizomeDB. RhizomeDB does not impose any hierarchy upon the entities, hence the name Rhizome, which is a subterranean plant stem that sends out roots and shoots from its nodes.

To access entities as objects which implement properties from interfaces like the examples above, RhizomeDB provides an API. For example, we can get a document based on the given file address as follows:

val document = lookupOne(DocumentFileEntity::fileAddress,

val document = lookupOne(DocumentFileEntity::fileAddress, fileAddress)?.document

val document = lookupOne(DocumentFileEntity::fileAddress,
                         fileAddress)?.document

Now the document object implements the DocumentEntity interface and we can use it to access the content of the document loaded in Fleet.

Our entity data model is flexible enough to represent not only data but also the data model itself. Suppose we want to develop a plugin (we’ll discuss Fleet’s plugins later in this series). Loaded plugins form part of Fleet’s state. All plugins share some common data required to integrate seamlessly with the application. However, every plugin has its own state, described with its own data model. This is not a problem for RhizomeDB. We can represent the plugin’s data model with entities. When we load a plugin, we also load its data model as a new entity. As a result, Fleet’s state management system is ready to accept the plugin’s state data.

State as a set of triples

Although our API gives us objects to work with entities, we don’t store them as such. Instead, we represent them in triples: [entity_id, attribute, value]. We call these triples datoms (the term comes from the Datomic database, which we’ve modeled our data structures after). 

Suppose the entity id for some particular file referring to a document is 18, and the entity id for the corresponding document is 19. The data would be stored as triples:

  • [18 :type DocumentFile]
  • [18 :document 19]
  • [18 :fileAddress "~/file.kt"]
  • [18 :readCharset "UTF-8"]

Note that properties of interfaces become attributes of triples. There are also various attributes like :type with special meanings. Types of values depend on the types of properties. When referring to other entities, property values are IDs.

The seemingly primitive structure of triples is quite effective when it comes to looking up data. Our engine is able to return very fast answers to queries in the form of a mask: [entity_id?, attribute?, value?], where any component may be either present or missing. The result of a query is always a set of datoms, which satisfies the given mask.

For example, we can ask for all filenames of currently loaded document files:

[? :fileAddress ?]

Or we can look for entity_id, which corresponds to a file with the given name:

[? :fileAddress "~/file.kt"]

For the second query, thanks to the uniqueness constraint, there should be no more than one answer in the resulting set.

To make queries run fast enough, the RhizomeDB maintains four indexes (each implemented as a hash trie):

  • Entity | Attribute | Value
  • Attribute | Entity | Value
  • Value | Attribute | Entity
  • Attribute | Value | Entity

The lookup* family of functions from the RhizomeDB API operates on these indexes to find the corresponding triples and build resultant entity objects.

RhizomeDB is heavily inspired by Datomic but adds some new ideas like read-tracking and query reactivity, which work for our use case. These features help us to deal with state changes as we’ll see shortly.

What is the change?

There is almost nothing curious about an immutable state. Interesting things come up when we change something. We’d like to know what was changed in the state and which UI elements need to be updated. To deal with changes we’ve implemented the following three ideas:

  • We record what exactly was changed as the novelty of the change.
  • We track what readers are querying.
  • We determine which queries would give new results because of this change.

Let’s discuss these ideas and see how they work in Fleet.

Novelty values

Remember that we strive to be immutable whenever possible, so we are not allowed to mutate values. Remember also that our state has the form of a snapshot containing a set of triples with entity IDs, attributes, and their values, representing corresponding data entities. Instead of mutating attributes’ values, for any change, we produce a new state snapshot with a new value of an attribute we want to change. A change then is simply removing an old value and adding a new one. To rename a file, for example, we do the following:

- [18 :fileAddress "~/file.kt"]

+ [18 :fileAddress "~/newFile.kt"]

- [18 :fileAddress "~/file.kt"] + [18 :fileAddress "~/newFile.kt"]

- [18 :fileAddress "~/file.kt"]
+ [18 :fileAddress "~/newFile.kt"]

Note that these two operations must be executed inside a transaction. Otherwise, you would observe the state without a filename at all. Running such a transaction gives us a new state snapshot with a new filename.

As such, any change is just a set of removals and additions of datoms. A transaction may result in many such removals and additions for different entities and attributes. Moreover, the difference between two snapshots is also such a set. From the entity IDs and attributes in the changeset, we know precisely which state components have been changed during the transaction. These are called the novelty of the change. Once we’ve executed a transaction, we record these novelty values.

Read-tracking and query reactivity

We know that readers access data in the state via queries. Queries have the form of a mask. It’s easy to track all the masks from a particular function. Once we have this information for all of our functions, we can determine which functions depend on which mask.

After every change, we get its novelty values. If we go over all the masks queried, we see which queries are affected by the change. Thanks to read-tracking, now we know which functions are affected. Consequently, we can invalidate the UI elements that call these functions. This makes the UI reaction highly efficient.

We use read-tracking for more than just updating UI elements. It’s quite a general mechanism that allows for useful patterns in reactive programming. For example, if we have a function that queries state, we can easily turn it into an asynchronous flow. Whenever changes in the state affect the result of such a function, we emit a new element of the flow. We can also safely cache query results without the risk of having outdated cached values. Once the value is updated in the state, we’ll know that immediately.

Summary

In this part of our series on how we build Fleet, we’ve employed an epochal time model via a series of immutable snapshots and built smart data representation in order to maintain our state. Our data exist on two levels: as data entities convenient for developers to work with, and as triples suitable for efficient looking up. When we change something, we record what was changed, determine those who are interested in these particular changes, and cause them to update the corresponding UI elements.

Keeping this background in mind, now we’re ready to discuss the distributed nature of Fleet’s state and transactional mechanisms that allow us to change it in a consistent way. We’ll do just that in the next blog post of this series. Stay tuned!

Read the whole story
yqrashawn
60 days ago
reply
JetBrans new Fleet editor implemented a new RhizomeDB inspired by Datomic
Share this story
Delete

Learnings from 5 years of tech startup code audits

1 Share

While I was at PKC, our team did upwards of twenty code audits, many of them for startups that were just around their Series A or B (that was usually when they had cash and realized that it’d be good to take a deeper look at their security, after the do-or-die focus on product market fit).

It was fascinating work – we dove deep on a great cross-section of stacks and architectures, across a wide variety of domains. We found all sorts of security issues, ranging from catastrophic to just plain interesting. And we also had a chance to chat with senior engineering leadership and CTOs more generally about the engineering and product challenges they were facing as they were just starting to scale.

It’s also been fascinating to see which of those startups have done well and which have faded, now that some of those audits are 7-8 years ago.

I want to share some of the more surprising things I’ve internalized from these observations, roughly ordered from most general to most security specific.

  1. You don’t need hundreds of engineers to build a great product. I wrote a longer piece about this, but essentially, despite the general stage of startup we audited being pretty similar, the engineering team sizes varied a lot. Surprisingly, sometimes the most impressive products with the broadest scope of features were built by the smaller teams. And it was these same “small but mighty” teams that, years later, are crushing their markets.
  2. Simple Outperformed Smart. As a self-admitted elitist, it pains me to say this, but it’s true: the startups we audited that are now doing the best usually had an almost brazenly ‘Keep It Simple’ approach to engineering. Cleverness for cleverness sake was abhorred. On the flip side, the companies where we were like ”woah, these folks are smart as hell” for the most part kind of faded. Generally, the major foot-gun (which I talk about more in a previous post on foot-guns) that got a lot of places in trouble was the premature move to microservices, architectures that relied on distributed computing, and messaging-heavy designs.
  3. Our highest impact findings would always come within the first and last few hours of the audit. If you think about it, this makes sense: in the first few hours of the audit, you find the lowest-hanging fruit. Things that stick out like a sore thumb just from grepping the code and testing some basic functionality. During the last few hours, you’ve fully contexted in to the new codebase, and things begin to click.
  4. Writing secure software has gotten remarkably easier in the last 10 years. I don’t have statistically sound evidence to back this up, but it seems like code written before around 2012 tended to have a lot more vulnerabilities per SLOC than code written after 2012 (we started auditing in 2014). Maybe it was the Web 2.0 frameworks, or increased security awareness amongst devs. Whatever it was, I think this means that security really has improved on a fundamental basis in terms of the tools and defaults software engineers now have available.
  5. All the really bad security vulnerabilities were obvious. Probably a fifth of the code audits we did, we’d find The Big One – a vulnerability so bad that we’d call up our clients and tell them to fix it immediately. I can’t remember a single case where that vulnerability was very clever. In fact, that’s part of what made the worst vulnerabilities bad — we were worried primarily because they’d be easy to find and exploit. “Discoverability” has been a component of impact analysis for a while, so this isn’t new. But I do think that discoverability should be much more heavily weighted. Discoverability is everything, when it comes to actual exposure. Hackers are lazy and they look for the lowest-hanging fruit. They won’t care about finageling even a very severe heap-spray vulnerability if they can reset a user’s password because the reset token was in the response (as Uber found out circa 2016). The counterargument to this is that heavily weighting discoverability perpetuates ”Security by Obscurity,” since it relies so heavily on guessing what an attacker can or should know. But again, personal experience strongly suggests that in practice, discoverability is a great predictor of actual exploitation.
  6. Secure-by-default features in frameworks and infrastructure massively improved security. I wrote a longer piece about this too, but essentially, things like React default escaping all HTML to avoid cross-site scripting, and serverless stacks taking configuration of operating system and web server out of the hands of developers, dramatically improved the security of the companies that used them. Compare this to our PHP audits, which were riddled with XSS. These newer stacks/frameworks are not impenetrable, but their attackable surface area is smaller in precisely the places that make a massive difference in practice.
  7. Monorepos are easier to audit. Speaking from the perspective of security researcher ergonomics, it was easier to audit a monorepo than a series of services split up into different code bases. There was no need to write wrapper scripts around the various tools we had. It was easier to determine if a given piece of code was used elsewhere. And best of all, there was no need to worry about a common library version being different on another repo.
  8. You could easily spend an entire audit going down the rabbit trail of vulnerable dependency libraries. It’s incredibly hard to tell if a given vulnerability in a dependency is exploitable. We as an industry are definitely underinvesting in securing foundational libraries, which is why things like Log4j were so impactful. Node and npm were absolutely terrifying in this regard—the dependency chains were just not auditable. It was a huge boon when GitHub released dependabot because we could for the most part just tell our clients to upgrade things in priority order.
  9. Never deserialize untrusted data. This happened the most in PHP, because for some reason, PHP developers love to serialize/deserialize objects instead of using JSON, but I’d say almost every case we saw where a server was deserializing a client object and parsing it led to a horrible exploit. For those of you who aren’t familiar, Portswigger has a good breakdown of what can go wrong (incidentally, focused on PHP. Coincidence?). In short, the common thread in all deserialization vulnerabilities is that giving a user the ability to manipulate an object that is subsequently used by the server is an extremely powerful capability with a wide surface area. It’s conceptually similar to both prototype pollution, and user-generated HTML templates. The fix? It’s far better to allow a user to send a JSON object (it has so few possible data types), and to manually construct the object based on the fields in that object. It’s slightly more work, but well worth it!
  10. Business logic flaws were rare, but when we found one they tended to be epically bad. Think about it — flaws in business logic are guaranteed to affect the business. An interesting corollary is that even if your protocol is built to provide provably-secure properties, human error in the form of bad business logic is surprisingly common (you need look no further than the series of absolutely devastating exploits that take advantage of badly written smart contracts).
  11. Custom fuzzing was surprisingly effective. A couple years into our code auditing, I started requiring all our code audits to include making a custom fuzzers to test product APIs, authentication, etc. This is somewhat commonly done, and I stole this idea from Thomas Ptacek, which he alludes to in his Hiring Post. Before we did this, I actually thought it was a waste of time—I just always figured it was an example of misapplied engineering, and that audit hours were better spent reading code and trying out various hypothesis. But it turns out fuzzing was surprisingly effective and efficient in terms of hours spent, especially on the larger codebases.
  12. Acquisitions complicated security quite a bit. There were more code patterns to review, more AWS accounts to look at, more variety in SDLC tooling. And of course, usually the acquisition meant an entirely new language and/or framework with its own patterns in use.
  13. There was always at least one closet security enthusiast amongst the software engineers. It was always surprising who it was, and they almost always never knew it was them! As security skillsets get more software-skewed, there’s huge arbitrage here if these folks can be reliably identified.
  14. Quick turnarounds on fixing vulnerabilities usually correlated with general engineering operational excellence. The best cases were clients who asked us to just give them a constant feed of anything we found, and they’d fix it right away.
  15. Almost no one got JWT tokens and webhooks right on the first try. With webhooks, people almost always forgot to authenticate incoming requests (or the service they were using didn’t allow for authentication…which was pretty messed up!). This class of problem led to Josh, one of our researchers, to begin asking a series of questions that led to a DefCON/Blackhat talk. JWT is notoriously hard to get right, even if you’re using a library, and there were a lot of implementations that failed to properly expire tokens on logout, incorrectly checked the JWT for authenticity, or simply trusted it by default.
  16. There’s still a lot of MD5 in use out there, but it’s mostly false positives. It turns out MD5 is used for a lot of other things besides an (in)sufficiently collision-resistant password hash. For example, because it’s so fast, it’s often used in automated testing to quickly generate a whole lot of pseudo-random GUIDs. In these cases, the insecure properties of MD5 don’t matter, despite what your static analysis tool may be screaming at you.

I’m curious if you’ve seen any of these, as well as others! Or, drop me a note if you disagree!

Read the whole story
yqrashawn
84 days ago
reply
Share this story
Delete

For a while, all ATC message routing in Germany was done through Emacs (2021)

1 Share

Article URL: https://teddit.net/r/emacs/comments/lly7po/do_you_use_emacs_lisp_as_a_general_purpose/gnvzisy/#c

Comments URL: https://news.ycombinator.com/item?id=31253981

Points: 1

# Comments: 0

Read the whole story
yqrashawn
145 days ago
reply
Share this story
Delete

Brave Clojure Jobs Blog -

1 Share

Learning any programming language requires a significant investment in time and resources. Of all the languages I’ve gained proficiency in, Clojure has by far yielded the largest benefits for my life. Some of these benefits have been surprising and have only become evident with time. I want to share these less-obvious benefits because they’ve had such a positive impact on me, and I want other devs to benefit, too.

Clojure’s stability can improve your lifestyle

Clojure’s core team is committed to the language’s stability over time so that code you wrote years ago will almost always work with the latest version of Clojure.

By unfortunate contrast, some other languages introduce significant, breaking changes like a friend declaring that they’re DONE with carbs and are going PALEO thank you very much. This imposes a cost every time you sit down to do some work. Say you want to spend some time on a side project that uses the language blub v1.0 but you want to use a library built for blub v1.1. You are in for a night of pain and horror.

Let’s play this scenario out over time. Say you only have 30 minutes to work on a side project. You run into a “language overhead” issue like the one above. Solving this kind of problem is chaotic, meaning it’s hard to pick up where you left off between programming sessions. If you don’t solve it the first time, you’ll retread a lot of ground the next session. Heaven help you if Life Happens and you can’t pick up your project for a couple weeks. It’s hard to make actual progress.

What does this have to do with your lifestyle? I’ll use this job board as an example. I recently re-built and re-launched this job board, and since then revenue has quintupled (!!!). This has provided more income, which obviously helps on the financial end. But what’s interesting is that Clojure’s stability has let me consistently make real progress on my passion projects without neglecting other parts of my life: making art, spending time with loved ones, etc. I can work less for better results because I’m not bogged down by the overhead costs imposed by the language.

I get to build this business at a sustainable pace without sacrificing my personal life because Clojure lets me focus on actually building instead of traversing the seven circles of language hell. I built some parts of the site eight years ago. The fact that I can figure something out and it’s still useful to me nearly a decade later is the definition of a good investment.

There’s another way that Clojure’s stability improves your lifestyle: it reduces your stress. The quality of your life is the quality of your day-to-day experience, and if a significant portion of that involves meaningless struggle with your programming language, well it’s hard to not end up feeling kinda lousy. And in this economy??

I realize I’m probably giving off strong “wow this guy is straight up mainlining the kool-aid” vibes here. Clojure, ultimately, is just a tool, right? Yes. And if we’re going to talk about building a satisfying and rewarding life, we need to talk about tools, and Clojure is an exceedingly good one.

Clojure is a great portal to other programming domains

Clojure has provided a gateway for me to learn more programming techniques and concepts, including:

There are other domains I want to explore, and great Clojure libraries for them:

You might look at this list and object that plenty of other languages offer libraries that let you explore other programming domains. In fact, some of the Clojure libraries I linked above are wrappers for Java libraries. So what makes Clojure special here?

Clojure has three properties that make it a superior language for learning programming concepts:

  1. Its REPL provides a tighter feedback looper than other languages, making it easier to perform mini experiments and learn from them. This is a form of self-testing, which studies show is one of the most effective tools for learning. You ask yourself a question, answer it, and compare it to the real answer, confirming (or not) your mental model of a system. The REPL allows you to do this almost at the speed of thought.
  2. Its focus on a small core set of data types and abstractions reduces the amount of non-essential learning you have to work through. In other languages, libraries introduce their own bespoke types with their own bespoke APIs, and the result is that you continually have to revisit the questions of “How do I represent data?” and “How do I transform it?”

    By contrast, 90% of the time the Clojure libraries you use represent compound data with vectors (which are like arrays), maps (like dictionaries), and sets. Even when they don’t, the data types they introduce likely participate in Clojure’s core abstractions, allowing you to use Clojure’s core functions. You don’t have to learn a new API for dealing with something like a LogicProgrammingSet or DiscreteEventMap.

  3. Its minimal syntax and lack of boilerplate. Less boilerplate means less ways to mess up, and less time writing boilerplate means more time experimenting and learning.

More than other languages, Clojure lets you focus on what’s essential about the domain or concept you’re trying to learn.

Your experience is portable across environments

Clojure was designed from the beginning to be platform-agnostic, and the result is that is that it’s made its way beyond the JVM to browser programming via ClojureScript, and to shell scripting via babashka. Being able to transfer your programming language experience from one environment to another like this means that you get to spend more time solving real problems.

It is hard to overstate how powerful this is. There are two complementary ways to think about this:

  1. You’re not limiting yourself to building only certain kinds of applications
  2. You’re gaining immense leverage

Generally, when you spend time gaining deep expertise in a programming language you’re necessarily limiting yourself to only building certain kinds of applications. Most languages are intimately tied to the kind of environment they target; when you spend time learning Go, you’re limiting yourself to server-side apps. When you learn Swift, you’re limiting yourself to iOS applications. If you want to start building a different kind of application, you have to learn a new language, with its attendant build tools and architecture ecosystem and paradigm and quirks. On top of that, you have to learn about the environment itself: its resources, its interaction modes, etc.

When you invest the time to learn Clojure, you gain leverage instead of limitations. Learning any language involves more than just the basics of syntax and build tools. It includes deeper topics like how to structure an application for maintenance and evolution. Taking the time to develop Clojure expertise will pay dividends when you switch from backend to frontend development.

I expect this situation will only improve over time, especially thanks to the unstoppable force that is Michiel Borkent, aka borkdude. I am very excited for what the future holds!

These Clojure Companies Hire People Without Clojure Experience

Are you ready to invest in Clojure? Then these companies are ready to invest in you. These businesses are successfully leveraging Clojure’s power, and they hire people without Clojure experience:

  • Nubank, the largest financial services company in Latin America and one of the biggest IPOs of 2021, is the world’s biggest user of Clojure and Datomic, with over 1000 Clojure developers.
  • Reify Health, a unicorn startup helping pharma companies and research sites enroll patients in clinical trials faster than ever before. Clinical research remains a significant bottleneck on drug development. Much of this is due to the slow and unpredictable nature of patient enrolment. Many potential therapies get scrapped because they failed to enroll enough patients in their trials. Reify Health is tackling this problem with a platform built on Clojure.
  • Pitch, the modern presentation software we always wished we had. Built for teams that care about where their time and energy goes. They’re around 80 Clojure engineers now, and are having a great time learning how to scale a Clojure codebase and team. They’ve got a good track record of hiring non-Clojurists and making them fluent in parens.
  • Metabase, the easiest way for people to get insights from their data, from tiny startups who get up and running quickly to major corporations with tens of thousands of users. Their codebase is open-source, and it’s one of the largest open-source Clojure codebases on the planet!
  • Logseq, a startup that exists to increase the knowledge output of humanity. They’re starting with building a personal knowledge assistant.
  • Mobot, a startup building developer tooling to help automate the manual side of mobile app QA (all the onerous work that remains after you’ve put in the time to leverage emulators and simulators). They’re using robots we fabricate to execute tests on a corpus of test devices. Their web apps and internal mobile app use ClojureScript.
  • Crossbeam, an escrow service for data with more than 5000 companies and more than $100M of venture capital. Their platform allows companies to find overlapping customers and prospects with their partners while keeping the rest of their data private and secure.
  • JUXT, a consultancy using Clojure to build systems and keep complexity under control. They’re hiring experienced Clojure practitioners as well as those that are keen to learn. JUXT also created XTDB and many well-known Clojure libraries.
  • BroadPeak Partners, a company that helps less technical users manage data streams and integrations without having to rely on developers. They’re focused on enterprises with increasing amounts of data to manage, the need to move fast, and deliver sustainable solutions.
  • Riverford Organic Farmers, an employee-owned company that delivers organic food to around 90,000 homes and businesses across the UK!
  • Shortcut.com, an intuitive and enjoyable project management platform, has been using Clojure for its backend since day one.
  • Dewise, a Cloud-first, polyglot, product and solutions development company, using Clojure for tricky data manipulation, tricky logic programming and dynamic business logic backends.
Read the whole story
yqrashawn
185 days ago
reply
Share this story
Delete

Write plain text files

1 Comment
2022-03-02

I write almost everything important in my life: thoughts, plans, notes, diaries, correspondence, code, articles, and entire books.

They are my extended memory — my noted self — my organized thoughts. I refer to them often. I search them, update them, and learn from them. I convert them into HTML to make websites, or LaTeX to make books.

My written words are my most precious asset. They are also a history of my life. That’s why I only use plain text files. They are the most reliable, flexible, and long-lasting option. Here’s why.

PORTABLE

I’ve brought my text files with me since 1990, from Mac to Windows to Linux to BSD, from PCs to laptops to tablets to Android to iOS to a tiny device the size of my thumb, and back again.

Every device, including ones long gone, and ones not invented yet, can read and edit plain text. Whether future virtual reality, or a chip you can implant in your earlobe, plain text will be there. Will Microsoft Word? Evernote? Notion? Maybe. Maybe not.

But plain text? Always. Everywhere.

UN-COMMERCIAL

Every few years a new company says you should use their special format. You have to pay them a monthly fee to use it — or keep all of your documents in their care. They offer some convenience or features, but at the cost of flexibility, portability, and independence.

When you store your writing in one company’s unique format, then you need that program to access it. Then the economy takes a turn, they go out of business, and your work is trapped in an unusable format.

You will outlive these companies. Your writing should outlive you. Depending on companies is not an option.

Plain text is un-commercial. It removes you from the world of subscriptions and hype. There will always be plenty of free, non-commercial software in the public domain for reading and editing text files.

OFFLINE

There are places and times when you can’t get online. Don’t depend on any tool that needs an internet connection.

There are great benefits to being intentionally offline and unreachable, to focus. It’s a super productivity boost. You need to be able to write, and have access to all your writing, during these times.

NO DEPENDENCIES

If you rely on Word, Evernote or Notion, for example, then you can’t work unless you have Word, Evernote, or Notion. You are helpless without them. You are dependent.

People tell me about more tools I could use in addition to my text files. But I don’t need or want anything else. Plain text files and a basic text editor are enough. This is everything you need for great thinking and writing. (A paper notebook and pencil are enough, too.)

If you only use plain text, you can work on any device, forever. The less you depend on, the better. Peace and focus come when you stop looking for more.

EASIEST TO CONVERT

Plain text can be converted into anything else.

HTML, Markdown, JSON, LaTeX, and many other standard formats, are just plain text. I’ve written four books and four hundred blog posts in plain text.

You can make your own personal formats in your plain text files. Maybe in each diary entry, the first two lines are like:

date: 2022-02-28
tags: where-to-live, kids, dog, anxious

Then it’s easy to use any little scripting language like Ruby, Python, or JavaScript to grab the date and tags, and use them for categorizing, sorting, renaming, archiving, or exporting.

Or if you don’t want to do it yourself, then it’s easy to find someone who can. Anyone who’s been programming for more than a week should be able to do it easily.

NEED HIERARCHY?

Use directories — also known as folders. These are also good for keeping your text together with other files like images and audio.

Documents/
Documents/Diary/
Documents/Diary/2022/
Documents/Diary/2022/2022-02-28.txt
Documents/Thoughts/
Documents/Thoughts/WhereToLive/
Documents/Thoughts/WhereToLive/2019-06-30.txt
Documents/Thoughts/WhereToLive/2020-01-18.txt
Documents/Ideas/
Documents/Ideas/MusicalChairs.txt
Documents/Ideas/NewHouse/
Documents/Ideas/NewHouse/Design/
Documents/Ideas/NewHouse/Design/entryway.jpg
Documents/Ideas/NewHouse/Design/roof.jpg
Documents/Ideas/NewHouse/Architect/
Documents/Ideas/NewHouse/Architect/JM_Lim.txt
Documents/Ideas/NewHouse/Architect/TPS_Inc.txt

NEED VISUALS OR GRAPHICS?

Need visual mind-mapping with circles and lines? Maybe you do. But maybe you don’t. Maybe it’s just another distraction, focusing on the tools instead of your thinking.

I love that plain text files have no formatting to tinker with. A tab key, SHIFT KEY, and vertical line breaks can go a long way, keeping you writing instead of formatting.

If you really need graphics, do your drawing using something else. Digital drawing into SVG files. Paper drawing, scanned into JPGs.

Formats that aren’t owned by any company. Formats that will outlast you.

Keep your graphics files alongside your text files. But keep your text as plain text.

CONCLUSION

Reliable, flexible, portable, independent, and long-lasting. Plain text files will be readable by future generations, hundreds of years from now.

I especially enjoy the tranquility of their offline, non-commercial nature. They’re quiet. They’re focused. (As I aim to be.)

screen shot of the text of this post
Read the whole story
yqrashawn
196 days ago
reply
And you can just copy paste it anywhere without any surprise
Share this story
Delete

Clojure - First Impressions

1 Share
Read the whole story
yqrashawn
197 days ago
reply
Share this story
Delete
Next Page of Stories