Agreed except for that last bit. Even if the session drags on endlessly the AI keeps using the same repetitive speech patterns (at least IME). Not only that but they stand out to me in that they don't match the nature of the conversation. It's a bit like receiving highly polished sales copy in response to an offhand forum post.
But importantly for dice we do understand the overarching principles that give rise to this. And dice don't output coherent sentences. Meanwhile in LLM land the analogous "roll of the dice" can result in a coherent response in natural language.
If you use a loaded dice, you can be pretty confident about where it will lands. It may not be 100% accurate, but can be quite close to certain. Without training the weight are pure noises. After training, it leans towards coherent sentences and particular statements.
Yes, and I believe my point still stands. We thoroughly understand the principle by which a loaded die can be intentionally biased despite not being able to predict the outcome of any given throw due to the system in question being a chaotic one.
In contrast, we do not understand LLMs in the same way (nor biological brains). Claiming that anything of that nature is simply biased towards coherent output seems entirely reductive to me - the question is how such coherence arises in the first place. There is no meaning encoded or computation performed by the particular pathway a die travels through the chaotic landscape.
Sure an argument can be made that it's "just" a next token predictor thus how is it really any different from a markov model? Yet the output is not even remotely the same.
> In contrast, we do not understand LLMs in the same way
From my point of view, (not a ML researcher), it’s due to the magic of numbers. The same thing happens with computer vision and neural networks. There’s a bunch of magic weights that get created which has no meaning by themselves, but computing them does help with detecting objects.
So if you take words, derives them into tokens, use the attention techniques to extract the “coherency” aspect, it’s no wonder you can replicate “coherency”. Add reinforcement learning to that to increase towards certain aspects like correct code syntax and you have heavily loaded the dice again.
We have used maths to model chemistry, biology, and physics, as well as economics and sociologic phenomena. Then we use maths (more specifically logic and set theory) to usher in the age of information and computing. Now you want us to act surprised that maths, through ML, can model language.
Maybe further down the line, we can have a simpler set of formulas for language coherency, but for now we have to make to with using the whole internet and a bazillion watts of power to guess the weights for the generic ML model.
I'm comparing it with chess. Chess is pretty complex, complex enough that only a small subset of humans can play it at a very high level. Introducing computers to chess first led to a statistical and brute force approach. But once that had paid off and the results were in people spent a lot of time analyzing those results and this led to an entirely new class of engine that was far more efficient than what had gone before and which performed even better than the 'big iron'.
I would not be surprised at all if we will find that AI will go the same route. The fact that we don't know how it works is where the opportunity for improvement lies.
No, they didn't. UB is a cop out and inserting yield is just plain bad. Locking up one or more threads in an implementation defined manner would be the outcome of least surprise (I already know it's going to lock up at least the one thread).
The standards intended interpretation of UB was always intended to be something like "implementation defined, no documentation required" to allow for implementation weirdness, even unpredictable ones. It was compiler authors who decided do abuse this allowance to do really unintuitive things instead of weird platform weirdness.
Because one can bleeping see that that's what would happen. Locking up a thread isn't a good thing, but it's a lot better than UB. There was never a need to make this UB.
I don't totally agree with this. To me, UB is an order of magnitude worse than pretty much anything else, so this is more than "slightly less" horrible. I don't necessarily disagree that this is still horrible, but I also don't write an C++, so I'm mostly just commenting as an outside observer.
It's only an incentive to start caring if it's cheaper than circumventing the law. In other words it won't work unless the aforementioned liability loophole is closed.
To rephrase the comment you replied to, if being a cowboy is more profitable (by whatever shady means) then that will generally be preferred by the market. Despite whatever sensibilities you or I might have there is no escaping that simple truth of capitalism.
Setting aside whether or not you can pull off a lock free approach here we can be certain of a couple things. There will be at least some overhead that must be paid somewhere even if that's on a separate management thread. And there will be a lot of additional complexity because that's just how concurrency always is.
Meanwhile the better the scheduler performs the more competitive the thread local approach becomes.
cost for interruption in an rseq critical section is that the PC gets overwritten to the rseq abort entry point before the task is rescheduled. no management thread necessary.
should be fairly minimal cost, especially assuming interruptions in the critical section are rare.
That's certainly interesting but I don't see how it would change my answer to you. Your question was why projects don't switch. My answer was because doing so seems likely to be a wash at absolute best.
Giving it some more thought, I expect caches will typically be wiped out by a context switch. So the only place rseq is likely to benefit an allocator is on systems with multiple NUMA nodes where you'd like to make sure any management code isn't paying a penalty by hitting the wrong address range.
IIUC rseq (ie CPU local data) is primarily good for two things. The first being obviating the need for atomics (specifically the resultant cache line ping-pong) but thread local data already accomplishes that. The second being massive oversubscription of physical CPU cores (ie tens of thousands of threads) where TLS becomes utterly wasteful while also thrashing the cache.
Are you sure you don't just have an axe to grind? Because that is an exceedingly uncommon edge case. Typically you only have a few threads and you aren't inundating the allocator with requests thus it is unlikely to make any practical difference.
If we do decide to concern ourselves with performance TLS has zero overhead and doesn't suffer from contention while rseq (at minimum) carries a penalty if preempted and involves setting a flag plus exhibits a data dependence for the address offset (the latter since AFAIK compilers don't natively support it as they do TLS). So while I'm certainly open to benchmarks to me it very much looks like a mixed bag that only comes up when you're already in questionable territory to begin with. In the event that we do step outside the norm I'd guess that a handful of threads with contention is a much more common scenario than thousands of threads per physical core exhibiting only minimal preemption.
I also expect something like a web server servicing thousands of requests in parallel to use an event loop instead of spawning an equivalent number of threads. I'm struggling to come up with a scenario where you haven't already fatally shot yourself in the foot and this remains a useful optimization to make. It's certainly relevant if you're using fibers (green threads, whatever you want to call them) but at that point you aren't in c calling malloc and your language runtime will (one hopes) already be taking care of all this for you.
> Are you sure you don't just have an axe to grind? Because that is an exceedingly uncommon edge case. Typically you only have a few threads and you aren't inundating the allocator with requests thus it is unlikely to make any practical difference.
Totally fair, and I agree that you should only have a few threads (or TPC) and shouldn’t be inundating the allocator with requests. But I’m only grinding this axe because I’ve personally had to deal with a system that went against most of that guidance.
We ran far too many threads in a memory-constrained environment. Thread count was many multiples of core count.
I totally agree that this is “questionable territory”, but honestly, any application that’s outgrown the basic glibc malloc has made a few mistakes.
reply