Plant startup is only the first step.[1] It's load pickup.
Here is a PJM training module on load pickup during system restoration.[2] It gives a sense of how touchy the process is. Power network control has good control over generation and transmission, but limited control over load.
When load is turned on, there are transient loads with different time constants. There's a huge load for the first second as inductors, capacitors, and incandescent filaments start up. That tails off in under a second. There's a second load as motors wind up to speed. Ten seconds or so. Then there's "cold load", where everything in HVAC starts trying to get temperatures back to normal. Maybe half an hour.
There's no mention of computer control. Listening to this, you visualize people running around reading meters and throwing big switches. It's probably people looking at display boards and sending commands to remote big switches, but the concept is the same.
Botching this means voltage or frequency goes out of tolerance, protective devices shut things down, and the system operators have to start over.
More PJM training modules on related subjects.[3]
Unclear what caused this yet. Something caused enough system instability to trip protective devices, but there's no good info yet. Once everybody has a chance to compare all the logging data from different points, it will make more sense.
The groundwork for the blackout will inevitably come back to a buildout and reliance of renewable sources that do not have spinning mass that can do frequency synchronization.
There are costly means to compensate for the lack of spinning baseload but actually building these devices have been neglected, to no ones surprise.
Why would you need a spinning mass when you have an inverter which can produce a range of frequencies and outputs and it all boils down to coordination?
Looking at power generation reports it's the plants relying on inertia (like nuclear) which were disconnected and haven't been reconnected since:
At the time I'm writing this, there appears to be about equal (hydro + gas) = (solar + wind). It appears they are running a significant portion of the grid with high inertia generators.
In a control system when you have an actuator that is much more responsive than the load, you can get into instable operating regimes where. On a small scale, when you are setting up servo motors, there is an inertia matching ratio for stability. In some of the systems I set up it was 10:1 load inertia:motor inertia. If you exceed that, you end up with elasticity between the motor and load that the motor can locally be above set point, and the control tells the motor to slow down, while the load has not yet reached set point. The motor is then too slow, and the control tells it to go faster, and you get the motor oscillating around the speed of the load.
On the grid you can have similar elasticity. An inverter many km away can get too far ahead in the AC cycle, then when that signal reaches the local inverter, the local inverter slows down, but the remote inverter has already slowed down, and by the time that slower wave gets to the local inverter, it is already too slow, and so the local inverter starts to speed up. If you get the frequency "right" you can end up with a positive feedback loop. With a high inertia rotating generator, the system is damped and slow enough to respond you don't change set point faster than the signal can propagate.
Seems like a simple cybernetics problem, especially if it's point-to-point. If there's many different inverters on the same circuit, you might need something like a directed graph, but there should be simpler, more local solutions.
I'm sure the inverter manufacturers are working on something.
At the start of solar and wind integration, it didn't matter. Large rotating generators were the main source on the grid, and provided stability. We're in a different world with an all-inverter grid control perspective-wise.
I think there is also still an information issue. The grid is definitely not point-to-point. It also isn't all generators. You also have loads switching on and off at indeterminate times. There is an overall predictability to it, but not at the phase governing resolution. If you have a local inverter and the phase slows down, you don't know if it is a remote inverter that has over-slowed, to which you shouldn't respond as it will fix itself, or a near-by load that suddenly switched on, to which you should respond as demand is now greater, and needs more supply. With big rotating generators, it doesn't really matter, as they are time averaging with the large inertia to stabilize the load/generator balance.
If the response rate is the only problem, you could just add damping to the system with the faster capability (especially when it is already software driven electronics).
You have a fundamental misunderstanding of both how a grid in blackout mode appears to a lonely inverter and what means of coordinating inverters that are available to said inverter.
Imagine you're trying to coordinate a choir of ten million singers that are scattered across a radius of a thousand kilometer, that all sing into their individual Ham radio. And you need it to come out in a perfect unison that sounds good for those who tune into the mixed broadcast.
"A problem of coordination" is perhaps correct, but it neglects the difficulties involved
Thing is, it's not ten million - it's several orders of magnitude fewer, particularly if we're starting with just a selected list of generators.
Also we know the locations of the generators, so we can calculate the phase shift and control them accordingly. At thousand kilometres it's around 60 degrees anyway, so not catastrophically huge.
But also, it isn't just generators, it is also the indeterminate loads. If you have nearby loads switching on and off that are similar sized to your local inverters, that is a complicated controls modelling problem.
I'm pretty sure in grid terms, 60 degrees out of sync is catastrophic. I'm pretty sure you'll be disconnected by protection circuits before it gets that far out.
Superluminal information transfer would be more useful for other areas than grid synchronization, but yeah we don't have crazy tech like that. And if we attempt to synch to a 50hz frequency over the internet means a 10ms latency difference results in 180 degree out of phase input.
And even if we did have the superluminal grid sync signal the propagation of electricity itself is slower than C and you'll need to consider perceived phase synchronization over distance.
Not my area of expertise, but I think this is wrong? Modern PV inverters used in rooftop solar could easily start frequency synchronization autonomously, but are regulated to not do that autonomously in (at least) parts of the EU.
More importantly, according to the REE, PV played a mayor role in the current and successful black start.
If the power is out, do you imagine that all the load that is out there disappears too?
If I'm in a blackout and put my multimeter into the grid socket, will the measurement be something that my local inverter can drive an AC load through with no problems?
And if that's not possible, how do we get an AC wave out to my 5000 local solar panel neighbors? And if we don't get any carrier wave to synch it to, then how do all these thousands of individual inverters decide on the unison synchronization necessary to all start jumping together?
And if they try and fail, will all the connected load accept a graceful out of synch mixed power noise at 0-220v until things properly latch together?
The problem isn't like playing a sine tune of acoustic sound that's merely audible to your local neighbor, who can dial in his own tune, and then daisy chaining this until it matches on a grid sized level.
I've always wondered why we can't just have a "master" power station transmit a 50Hz signal modulated at, say, 50MHz. Then all the other station simply sync to that, including small scale such as domestic PV etc.
Then depending on how out-of-sync a given station is with the signal, they ramp production up or down accordingly.
Is the problem here that there is inadequate or slow feedback to the supply side regarding load and therefore you'd end up with under-voltage?
That's more or less how it works. Knowing what 50hz is and how out of sync a generator is is easy. AFAIK, the problem is one of inertia, both electrical and mechanical. A generator is a spinning mass, the more load on the generator, then the slower it turns. How fast it turns decides the frequency of power it generates. Say you get your generators running at 50hz at night, and then during the day it gets hot and thousands of AC units get turned on. Now your generators are running at 40hz.
It's not about getting the generator to spin at 50hz, it's about getting it to spin at 50hz under dynamic load. Most generators are mostly fixed speed I think for the sake of efficiency, so the mechanism the grid has to regulate the frequency of power is largely by controlling the number of generators running at any given time, and how much of the grid is connected to them. Ramping up and down based on the current observed frequency of the grid is exactly how this all works. However it's more complicated because what if two or more power stations see a low frequency grid and decide to ramp up more generators? Well now the grid frequency is too high. If they them respond to that by ramping down, it'll go too low. Rinse and repeat and you get undesirable frequency oscillation, so there needs to be more communication across the grid than just responding to the locally observed frequency.
Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart by the sheer inertia of the power grid forcing it to match speed and phase in an instant. This is also a problem when attempting to connect two isolated grids, of they're too out of phase with each other, catastrophic physical damage will occur.
Thanks for the reply. In the 50Mhz-modulated-50Hz scenario, however, the modulated 50Hz signal doesn't change under load; it has zero inertia (or infinite inertia maybe) because it's electronically generated.
Any individual station that gets out of sync either works hard to catch up, or gets disconnected.
The problem, as I see it, is that the load signal is equal and equivalent to the sync signal and maybe they could be separated. (e.g. the FM modulated signal wouldn't have to be a sine wave, nor would it even have to be 50Hz)
> what if two or more power stations see a low frequency grid and decide to ramp up more generators?
Seems to me it would solve this problem - they both have a single invariant sync signal.
> Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart
Seems to me this problem is solvable with electronics, but I don't know enough to say how much out-of-phase is problematic. What if, as you say, some heavy load is switched on, the local nuclear turbines slow by a fraction for a few 10s of seconds, and so a home PV array starts leading by 0.01Hz (e.g. home PV at 50.00Hz, local power station at 49.99Hz, so after 10s the PV is fully 36º or 0.6rad ahead). Does this result in the inverter (or the small local windfarm) exploding? Is that a realistic scenario?
> there needs to be more communication across the grid than just responding to the locally observed frequency
Precisely - a grid-wide sync signal, modulated at high frequency.
You need to consider that a master clock signal transmitted through air will arrive at a different time than an AC power wave transmitted through a wire.
A millisecond latency will cause a significant out of phase shift to the AC signal too.
The AC data you want is already codified in the power signal
Unfortunately that pretty much just leaves solar panels, as wind/hydro/concentrated solar/geothermal all rely on a spinning mass moving at a mostly consistent speed, though at least hydro can black start itself relatively easily
Just like there is vibe coding, this is vibe downvoting: “they are probably right, but kinda feels like they are dissing renewables, and I like renewables, better downvote just in case”
You're right, but: the renewables "debate" is a massive tar-pit of bad faith arguments. People are tired of rehashing that. The best solution is not to slow the rollout of renewables but to accelerate the rollout of battery storage.
(also we're a bit early for a proper postmortem, but that's an inevitable reality of social media, got to get the talking points in before the facts)
> The best solution is not to slow the rollout of renewables but to accelerate the rollout of battery storage.
I agree as well. And I think the gp (or the ggp?) post noticed that there is a lack of "spinning mass" to do frequency damping would have agreed as well. They could be merely suggesting to add just spinning mass synchronizers. These could be independent units, doing just that or maybe even convert some old power plants: "just" decouple the turbine from the generator.
For about 2-3 years the default for people here has been to downvote everything rather than driving discussion; much more reddit-esque than what it used to be. There are no other sites like this one to jump to, unfortunately.
On-topic: Wind power brings in about 20% of Spain's electricity, and that one fluctuates a lot indeed. I hope they put out a detailed post-mortem, I'd be an enjoyable read.
Sounds like these systems are extremely vulnerable to sabotage. But adding more complexity such as distributed digitally controlled loads will not necessarily make the situation much better.
Dropping from 32 to 8 GW usually means that an interconnected grid has fragmented into islands. If an island has a blackout, you need to black start it and resynchronise it with other islands.
But what prevents you from connecting the stopped power plants to one of those islands and then running it from there? I'm not claiming that is easy, but its not a black start.
It looks to me like the author of this article has fundamentally misunderstood what a black start is. The fact that some power generation remained both on the peninsula, but also on imported power being available means that to my understanding this cannot be classified as a "black start".
A lot of overtime, a lot of careful coordination yes, a black start, no
A black start is a process, not a situation. The situation they're in might not fully necessitate a black start, but a black start procedure might still be the best course of action. Some plants have that capability and the rest (should) have a direct tie to one that does. For all practical purposes, that's a black start.
Oh man, this reminds me of factorio. When you miscalculate your power needs and your generators eat up the fuel that isn't being produced enough. I've had to make a bootstrapping mechanisms and power switches for some of my powerplants to ensure I can get it back up in case it goes down and I can then re-enable my factory bit by bit.
Unfortunately, you can't simply hand-feed 100 coal into all the Spanish thermal plants and then plop down two more to be on the safe side until you've fixed supply...
it's that in Factorio there are no problems with phase synchronization, and the devices, consumers, producers, so the machines don't get damaged from a lot of abrupt starting/stopping.
still, it's strange that operators and grid controllers have to do the restart manually.
It's even worse in satisfactory imo, because the game is build around your fabs consuming wildly varying amounts of power throughout their production cycle... And after a power loss, they'll not only need the initial burst (can be handled via batteries) but will also now all be synchronized to have their power spikes at exactly the same moment.
Dyson sphere program is the by far most forgiving in that regard. You can overdraw to something like 200% before it becomes really problematic, and even that quickly resolves itself the moment you get more power online
Also in Satisfactory pumps need power to run, so coal power plants quite realistically won't start without extra power (unless you have a water tower on the input).
Yes, the huge rotating mass is inherently more stable than a diesel. They key to grid stability is mostly huge rotating mass, either producing or consuming.
Can't answer for aluminium, but I can answer for window glass: One of our plants was threatened by massive flooding 2021, threatening the float glass oven (which doesn't float itself despite the name). It was closed off with insulation, turned off, and everyone hoped for the best and no steam explosion. In the end, only the cellars with the control equipment flooded and it could be gradually (as the control cabinets were checked and restarted) brought up again without loosing too much heat a day later.
In short: Heat Inertia of large molten bodies is massive with good insulation. If the time is too long tho, only dynamite will dismantle a solid chunk of material again.
Its not an oven alone, its a kiln that smelts up the components for glass of the desired recipe, then purs the glass on a long (~half a km) basin of molten tin where it stretches and cools slowly, pushing the solidifying mass down the basin until the end. Re-Melting it is as far as i know not possible, as the melting is done before the pouring. You can melt the tin again, but not the glass.
Interesting! Does this mean that you could use intermittent energy sources, like solar power, to heat up the float glass oven during the day and it woul keep enough heat during the night so that it could keep running in the morning?
Float glass is a continuous process, so it runs 24/7 and too much product is recycled into the smelter at the start again (Funnily, glass waste is actually a scarce resource). Getting the process dialed in to emit the desired quality is a long winded thing as far as i know (not involved in the process myself). Also, the heating is done with gas (LNG/CNG/H2) and electric in combination.
The Insulation is in place all the time because the process is very energy intensive and the glass over the whole basin is supposed to cool evenly. Just the end is not closed off as the glass is ejected in a continuous endless stream[1].
In some cases they don't have any practical ability to.
The Texas winter storm resulted in a lot of scrap at semiconductor lines due to power loss. There are industries that are completely dependent on grid power. You can't generate enough on site to back up 100% of your operation. Think about how much power one EUV source consumes.
The only reason Samsung is building additional factories in Texas is because the local utilities are effectively treating them as a critical load. The new plants are right by the ERCOT operation center and likely have access to the same cranking paths that the grid uses for black starts.
Thanks for sharing the link! 1.3MW really isn’t that high of a load though. Thats a very straightforward load to be able to support via diesel generators. As an example, a typical large datacenter uses an order of magnitude more power, yet stays online following a grid outage.
Anecdotal example: The, now closed, Aluminium plant here was build only because the government build a nuclear plant, which is still going. There was a coal plant next to it (closed now) as a backup for when nuclear was not available.
Funny thing is most people thought the coal plant was the nuclear plant when driving by since it looks more like a plant, the nuclear dome structure is kind of hidden from the road and since natural water is used as cooling no big towers are close by.
I had the same experience living near a hospital. The whole city has gone out a few times, and our grid was either still up, or is one of the first grids back up.
That was a fun 10 hours or so that we were without power in most of Barcelona. Some barrios had ~some~ power but most of us were without.
The one thing that surprised me is how quickly rumors started about power being out in Portugal (true), France (true to an extent), Belgium (false), and the UK (false). Walking back home from work you heard the names of Trump being mentioned as well.
It's a bit scary to personally experience how quickly people can start panicking. On the other hand the bars that had some power had a great afternoon with tons of beer being served before it got too warm :-)
I am in Barcelona too. The combination of no power and patchy internet was the perfect recipe for rumors to spread, which was fun to observe. I went for a walk since I couldn't work and saw tons of people socializing, drinking beers (more than usual), and young people playing without their phones. It'll be a special day to remember, I think.
If we're ready for it and can prepare, it wouldn't be so bad I'd guess. Trouble yesterday was that we had no idea about the scope, I was in the middle of finishing moving houses, but was promptly held back when I arrived with a van to Barcelona and noticed it wasn't just my town that was without power.
I was actually considering going for a ride on the motorcycle today if the outage lasted another day.
Beautiful weather, a tank full of gasoline, and no stress about work? Sign me up!
Most people I spoke to around Maresme area yesterday believed someone incompetent made a mistake somewhere, and that was the reason. Some people jokingly suggested cyber attack/Russian probes, but afaik no one took those seriously.
A security guard at my workplace actually told me they had "received a call" about power being out in Romania as well. I'm not sure how that one rumour spread.
In my experience, shops you frequent will be sympathetic to that problem and will just let you come back to pay later. And I imagine that any shops that sell stuff that needs to be kept cool were especially amenable to that ...
How warm is "too warm"? I find cold beer, and cold soft drinks as well, quite repulsive. I usually try to order without ice entirely, and if I receive any bottles or cans, I permit them to come to room temp before consumption.
Refrigeration is sort of a scourge to those who enjoy food. We are so used to simply freezing our esophagus and not tasting anything! Food is kept cold because it lasts longer, not because it tastes better.
Sure, refrigeration makes restaurants and grocery stores possible. But it also presents challenges to cooking and it's very resource-intensive -- think about it -- basically everyone everywhere is always running their refrigeration -- that's a huge load on any system.
Pedantic, but most beers should be drank (according to the manufacturers and experts) at 7-13c depending on type of beer. Exceptions always exist to confirm the rule. Which is above fridge and around "cellar" temperatures.
Unless you drink Schultenbrau, which tastes worse cold than at room temperature (/s, kind of).
Do they? But why? Why would they? Are they simply accustomed to tasting it that way? I should say "not tasting it" because chilling a beverage basically kills its flavours, and they never make it to your tongue correctly.
I would say that many people would be surprised and amazed if they tasted such things as Pepsi without ice, and without coming right out of the fridge.
Unfortunately, any lack of refrigeration really messes up supply chains and will eat into anyone's bottom line, so it's not a realistic goal.
This stuff is panel-tested out of the wazoo. There's a reason bitterness, hoppiness, alcohol levels, differ market by market, even on brands of the same brand/line extension.
Beer also does not require refrigeration throughout the entire supply chain, they happily sit in a warm warehouse waiting to be picked up by an unrefrigerated truck, only to be cooled at the outlet or consumer.
I've always theorized that, in prehistory, the only source of cold water would be recent snowmelt; streams that came fairly directly from a probably-clean source. Warm water could have all sorts of things growing in it. So our ancestors who found cool water to be particularly refreshing and sought it out might have suffered less from waterborne illness.
I can only imagine the difficulties facing technicians and engineers black starting a nation-scale grid.
I operate a microgrid facility in Hispaniola and have wonderfully cooperative users backed by a separately powered communications and control system. Even for us, a facility serving a small neighborhood and farm, a black start must be performed as a careful choreography of systems and loads…and we can just pick up a radio and tell people to turn the main breaker of their house on or off, and to leave their AC units off until we finish bringing everything up. In 12 years we’ve only had to do it twice, but even for us it’s a tedious process.
I am no expert, however with solar power and more grid batteries (I don’t think Spain has many grid batteries) wouldn’t it be possible to charge the batteries and use that to drive a ‘black start’.
Sounds simple, and I appreciate from experience a huge amount of prep and validation needs to be done in the background.
At an extremely abstract level, it is conceptually simple, but most inverters are designed with safeguards to prevent exactly this. They only produce power when the grid is up, and in phase with it.
Because if not sequenced and interlocked, islanding behavior can result in back-powering parts of the grid that are supposed to be disconnected and down for maintenance, or creation of desynchronized islands that then cause more damage when reconnected.
This is absolutely a problem that operators need to solve. Radio synchronization is possible, neighborhood islanding could be beneficial, etc. There are entire village-wide micro-grids in remote places where everyone's inverters do perform these functions, it just hasn't been embraced in monopoly environments yet.
For instance, relays that close isolated parts back onto the main network would need to be phase-aware, to only reconnect if it's safe to do so. You could mandate that all island-mode inverters center themselves around 50.1 or 60.1Hz in the absence of outside influence so there's certain to be phasing opportunities. Distribution networks would need more switches to provide positive isolation during maintenance. And all of these things would need huge amounts of interoperability testing. It's possible, but it has a cost.
Initial estimates went from "No idea" to "Could take days".
In the end it took about 12 hours to restore most of the power in two countries. There are some knock-on effects from trains being in the wrong places, but a solid result overall.
This is not correct. The REE press conference held about two hours after the start of the blackout gave an estimate of 6 to 10 hours, which was about right. Any other estimations were misinformation.
What caused it? Nowadays when anything like this happens in the EU, I immediately wonder if it was Russian saboteurs.
They have certainly been destroying infrastructure here, eg. undersea internet and power cables.
I was wondering if a solar flare could cause something like this. But if it had been that on this occasion, the astronomers would've been warning about it for at least 8 minutes. :-)
Spain is mostly back up.
Plant startup is only the first step.[1] It's load pickup.
Here is a PJM training module on load pickup during system restoration.[2] It gives a sense of how touchy the process is. Power network control has good control over generation and transmission, but limited control over load.
When load is turned on, there are transient loads with different time constants. There's a huge load for the first second as inductors, capacitors, and incandescent filaments start up. That tails off in under a second. There's a second load as motors wind up to speed. Ten seconds or so. Then there's "cold load", where everything in HVAC starts trying to get temperatures back to normal. Maybe half an hour.
There's no mention of computer control. Listening to this, you visualize people running around reading meters and throwing big switches. It's probably people looking at display boards and sending commands to remote big switches, but the concept is the same.
Botching this means voltage or frequency goes out of tolerance, protective devices shut things down, and the system operators have to start over.
More PJM training modules on related subjects.[3]
Unclear what caused this yet. Something caused enough system instability to trip protective devices, but there's no good info yet. Once everybody has a chance to compare all the logging data from different points, it will make more sense.
[1] https://pjm.adobeconnect.com/_a16103949/p622tuwooba/
[2] https://pjm.adobeconnect.com/p6e5csm81ter/
[23 https://www.pjm.com/training/training-resources
The groundwork for the blackout will inevitably come back to a buildout and reliance of renewable sources that do not have spinning mass that can do frequency synchronization.
There are costly means to compensate for the lack of spinning baseload but actually building these devices have been neglected, to no ones surprise.
Why would you need a spinning mass when you have an inverter which can produce a range of frequencies and outputs and it all boils down to coordination?
Looking at power generation reports it's the plants relying on inertia (like nuclear) which were disconnected and haven't been reconnected since:
https://app.electricitymaps.com/zone/ES/72h/hourly
At the time I'm writing this, there appears to be about equal (hydro + gas) = (solar + wind). It appears they are running a significant portion of the grid with high inertia generators.
In a control system when you have an actuator that is much more responsive than the load, you can get into instable operating regimes where. On a small scale, when you are setting up servo motors, there is an inertia matching ratio for stability. In some of the systems I set up it was 10:1 load inertia:motor inertia. If you exceed that, you end up with elasticity between the motor and load that the motor can locally be above set point, and the control tells the motor to slow down, while the load has not yet reached set point. The motor is then too slow, and the control tells it to go faster, and you get the motor oscillating around the speed of the load.
On the grid you can have similar elasticity. An inverter many km away can get too far ahead in the AC cycle, then when that signal reaches the local inverter, the local inverter slows down, but the remote inverter has already slowed down, and by the time that slower wave gets to the local inverter, it is already too slow, and so the local inverter starts to speed up. If you get the frequency "right" you can end up with a positive feedback loop. With a high inertia rotating generator, the system is damped and slow enough to respond you don't change set point faster than the signal can propagate.
Seems like a simple cybernetics problem, especially if it's point-to-point. If there's many different inverters on the same circuit, you might need something like a directed graph, but there should be simpler, more local solutions.
I'm sure the inverter manufacturers are working on something.
At the start of solar and wind integration, it didn't matter. Large rotating generators were the main source on the grid, and provided stability. We're in a different world with an all-inverter grid control perspective-wise.
I think there is also still an information issue. The grid is definitely not point-to-point. It also isn't all generators. You also have loads switching on and off at indeterminate times. There is an overall predictability to it, but not at the phase governing resolution. If you have a local inverter and the phase slows down, you don't know if it is a remote inverter that has over-slowed, to which you shouldn't respond as it will fix itself, or a near-by load that suddenly switched on, to which you should respond as demand is now greater, and needs more supply. With big rotating generators, it doesn't really matter, as they are time averaging with the large inertia to stabilize the load/generator balance.
If the response rate is the only problem, you could just add damping to the system with the faster capability (especially when it is already software driven electronics).
bless you for your correct use of the C-word.
Again, this appears to be a problem of coordination. An inverter doesn't have to:
-react immediately
-use its full capacity all the time
You have a fundamental misunderstanding of both how a grid in blackout mode appears to a lonely inverter and what means of coordinating inverters that are available to said inverter.
Imagine you're trying to coordinate a choir of ten million singers that are scattered across a radius of a thousand kilometer, that all sing into their individual Ham radio. And you need it to come out in a perfect unison that sounds good for those who tune into the mixed broadcast.
"A problem of coordination" is perhaps correct, but it neglects the difficulties involved
Thing is, it's not ten million - it's several orders of magnitude fewer, particularly if we're starting with just a selected list of generators.
Also we know the locations of the generators, so we can calculate the phase shift and control them accordingly. At thousand kilometres it's around 60 degrees anyway, so not catastrophically huge.
But also, it isn't just generators, it is also the indeterminate loads. If you have nearby loads switching on and off that are similar sized to your local inverters, that is a complicated controls modelling problem.
I'm pretty sure in grid terms, 60 degrees out of sync is catastrophic. I'm pretty sure you'll be disconnected by protection circuits before it gets that far out.
Man, if only we had some means of instantaneous communication at a distance. What a crazy world that would be, huh?
Superluminal information transfer would be more useful for other areas than grid synchronization, but yeah we don't have crazy tech like that. And if we attempt to synch to a 50hz frequency over the internet means a 10ms latency difference results in 180 degree out of phase input.
And even if we did have the superluminal grid sync signal the propagation of electricity itself is slower than C and you'll need to consider perceived phase synchronization over distance.
Not my area of expertise, but I think this is wrong? Modern PV inverters used in rooftop solar could easily start frequency synchronization autonomously, but are regulated to not do that autonomously in (at least) parts of the EU.
More importantly, according to the REE, PV played a mayor role in the current and successful black start.
How?
If the power is out, do you imagine that all the load that is out there disappears too?
If I'm in a blackout and put my multimeter into the grid socket, will the measurement be something that my local inverter can drive an AC load through with no problems?
And if that's not possible, how do we get an AC wave out to my 5000 local solar panel neighbors? And if we don't get any carrier wave to synch it to, then how do all these thousands of individual inverters decide on the unison synchronization necessary to all start jumping together?
And if they try and fail, will all the connected load accept a graceful out of synch mixed power noise at 0-220v until things properly latch together?
The problem isn't like playing a sine tune of acoustic sound that's merely audible to your local neighbor, who can dial in his own tune, and then daisy chaining this until it matches on a grid sized level.
I've always wondered why we can't just have a "master" power station transmit a 50Hz signal modulated at, say, 50MHz. Then all the other station simply sync to that, including small scale such as domestic PV etc.
Then depending on how out-of-sync a given station is with the signal, they ramp production up or down accordingly.
Is the problem here that there is inadequate or slow feedback to the supply side regarding load and therefore you'd end up with under-voltage?
That's more or less how it works. Knowing what 50hz is and how out of sync a generator is is easy. AFAIK, the problem is one of inertia, both electrical and mechanical. A generator is a spinning mass, the more load on the generator, then the slower it turns. How fast it turns decides the frequency of power it generates. Say you get your generators running at 50hz at night, and then during the day it gets hot and thousands of AC units get turned on. Now your generators are running at 40hz.
It's not about getting the generator to spin at 50hz, it's about getting it to spin at 50hz under dynamic load. Most generators are mostly fixed speed I think for the sake of efficiency, so the mechanism the grid has to regulate the frequency of power is largely by controlling the number of generators running at any given time, and how much of the grid is connected to them. Ramping up and down based on the current observed frequency of the grid is exactly how this all works. However it's more complicated because what if two or more power stations see a low frequency grid and decide to ramp up more generators? Well now the grid frequency is too high. If they them respond to that by ramping down, it'll go too low. Rinse and repeat and you get undesirable frequency oscillation, so there needs to be more communication across the grid than just responding to the locally observed frequency.
Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart by the sheer inertia of the power grid forcing it to match speed and phase in an instant. This is also a problem when attempting to connect two isolated grids, of they're too out of phase with each other, catastrophic physical damage will occur.
Thanks for the reply. In the 50Mhz-modulated-50Hz scenario, however, the modulated 50Hz signal doesn't change under load; it has zero inertia (or infinite inertia maybe) because it's electronically generated.
Any individual station that gets out of sync either works hard to catch up, or gets disconnected.
The problem, as I see it, is that the load signal is equal and equivalent to the sync signal and maybe they could be separated. (e.g. the FM modulated signal wouldn't have to be a sine wave, nor would it even have to be 50Hz)
> what if two or more power stations see a low frequency grid and decide to ramp up more generators?
Seems to me it would solve this problem - they both have a single invariant sync signal.
> Connect a stopped, or sufficiently out of phase generator to a full scale power grid and the puny thousands of pounds of metal that make up a generator are going to get ripped apart
Seems to me this problem is solvable with electronics, but I don't know enough to say how much out-of-phase is problematic. What if, as you say, some heavy load is switched on, the local nuclear turbines slow by a fraction for a few 10s of seconds, and so a home PV array starts leading by 0.01Hz (e.g. home PV at 50.00Hz, local power station at 49.99Hz, so after 10s the PV is fully 36º or 0.6rad ahead). Does this result in the inverter (or the small local windfarm) exploding? Is that a realistic scenario?
> there needs to be more communication across the grid than just responding to the locally observed frequency
Precisely - a grid-wide sync signal, modulated at high frequency.
You need to consider that a master clock signal transmitted through air will arrive at a different time than an AC power wave transmitted through a wire.
A millisecond latency will cause a significant out of phase shift to the AC signal too.
The AC data you want is already codified in the power signal
Unfortunately that pretty much just leaves solar panels, as wind/hydro/concentrated solar/geothermal all rely on a spinning mass moving at a mostly consistent speed, though at least hydro can black start itself relatively easily
I don't see why. Battery-backed renewables should be much easier to synchronize than spinning generators.
Do you mean solar energy sources? Wind turbines have plenty of "spinning mass"...
Not sure why you are downvoted but this rings true to me.
Just like there is vibe coding, this is vibe downvoting: “they are probably right, but kinda feels like they are dissing renewables, and I like renewables, better downvote just in case”
You're right, but: the renewables "debate" is a massive tar-pit of bad faith arguments. People are tired of rehashing that. The best solution is not to slow the rollout of renewables but to accelerate the rollout of battery storage.
(also we're a bit early for a proper postmortem, but that's an inevitable reality of social media, got to get the talking points in before the facts)
> The best solution is not to slow the rollout of renewables but to accelerate the rollout of battery storage.
I agree as well. And I think the gp (or the ggp?) post noticed that there is a lack of "spinning mass" to do frequency damping would have agreed as well. They could be merely suggesting to add just spinning mass synchronizers. These could be independent units, doing just that or maybe even convert some old power plants: "just" decouple the turbine from the generator.
For about 2-3 years the default for people here has been to downvote everything rather than driving discussion; much more reddit-esque than what it used to be. There are no other sites like this one to jump to, unfortunately.
On-topic: Wind power brings in about 20% of Spain's electricity, and that one fluctuates a lot indeed. I hope they put out a detailed post-mortem, I'd be an enjoyable read.
Exhibit A: this comment.
Some even borrowed the essence of it, posted it elsewhere and still downvoted, ¯\_(ツ)_/¯.
Sounds like these systems are extremely vulnerable to sabotage. But adding more complexity such as distributed digitally controlled loads will not necessarily make the situation much better.
In theory maybe, but Ukraine are doing a great job of keeping their grid up in a hot war despite attacks on power infrastructure.
But according to production data it has never dropped to zero? Why it would be a black start?
It dropped from about 32GW to about 8GW
source: https://transparency.entsoe.eu/generation/r2/actualGeneratio...
though admittedly https://en.wikipedia.org/wiki/Black_start mentions that partial shutdown may be also requiring a black start?
Dropping from 32 to 8 GW usually means that an interconnected grid has fragmented into islands. If an island has a blackout, you need to black start it and resynchronise it with other islands.
But what prevents you from connecting the stopped power plants to one of those islands and then running it from there? I'm not claiming that is easy, but its not a black start.
It looks to me like the author of this article has fundamentally misunderstood what a black start is. The fact that some power generation remained both on the peninsula, but also on imported power being available means that to my understanding this cannot be classified as a "black start". A lot of overtime, a lot of careful coordination yes, a black start, no
A black start is a process, not a situation. The situation they're in might not fully necessitate a black start, but a black start procedure might still be the best course of action. Some plants have that capability and the rest (should) have a direct tie to one that does. For all practical purposes, that's a black start.
Oh man, this reminds me of factorio. When you miscalculate your power needs and your generators eat up the fuel that isn't being produced enough. I've had to make a bootstrapping mechanisms and power switches for some of my powerplants to ensure I can get it back up in case it goes down and I can then re-enable my factory bit by bit.
Unfortunately, you can't simply hand-feed 100 coal into all the Spanish thermal plants and then plop down two more to be on the safe side until you've fixed supply...
but sorta you can, that's not the hard part
it's that in Factorio there are no problems with phase synchronization, and the devices, consumers, producers, so the machines don't get damaged from a lot of abrupt starting/stopping.
still, it's strange that operators and grid controllers have to do the restart manually.
The machines might still end up being eaten by giant alien bugs due to the power outage shutting down automated defenses. :)
It's even worse in satisfactory imo, because the game is build around your fabs consuming wildly varying amounts of power throughout their production cycle... And after a power loss, they'll not only need the initial burst (can be handled via batteries) but will also now all be synchronized to have their power spikes at exactly the same moment.
Dyson sphere program is the by far most forgiving in that regard. You can overdraw to something like 200% before it becomes really problematic, and even that quickly resolves itself the moment you get more power online
Also in Satisfactory pumps need power to run, so coal power plants quite realistically won't start without extra power (unless you have a water tower on the input).
If you're interested in the process of a black start
https://www.youtube.com/watch?v=uOSnQM1Zu4w
from Practical Engineering
We had something similar happen in Tunisia in 2002. The reliance on diesel turbines meant a very long rebooting process.
As a result, the government invested heavily in installing gas turbines all over the country.
When we had another blackout event in 2018(?), it only took an hour or so to get things back again.
Why is that? Are turbines easier to speed regulate than diesel engines?
Yes, the huge rotating mass is inherently more stable than a diesel. They key to grid stability is mostly huge rotating mass, either producing or consuming.
Or I guess, by inference, huge battery banks.
Battery tech is an upstart in the grid stabilization world, the standard is actually huge flywheels https://www.gevernova.com/power-conversion/sites/default/fil...
I imagine a lot of sunshine in Tunisia (not too familiar with your country) , is there a big transition to solar?
Yes. Many laws have been enacted in the last 5 years to enable more solar project.
The downside is that the choice was made to focus on smaller 10-20 MW projects, with the occasional 100+ MW installation.
Hence, despite new projects entering service every couple months solar only accounts for less than 3% of electricity production nationwide.
How do energy intensive processes like aluminium smelters or window glass producers preven their machines from self destruction ?
Can't answer for aluminium, but I can answer for window glass: One of our plants was threatened by massive flooding 2021, threatening the float glass oven (which doesn't float itself despite the name). It was closed off with insulation, turned off, and everyone hoped for the best and no steam explosion. In the end, only the cellars with the control equipment flooded and it could be gradually (as the control cabinets were checked and restarted) brought up again without loosing too much heat a day later.
In short: Heat Inertia of large molten bodies is massive with good insulation. If the time is too long tho, only dynamite will dismantle a solid chunk of material again.
Is the heater keeping the material liquid not enough to re-melt it again?
Its not an oven alone, its a kiln that smelts up the components for glass of the desired recipe, then purs the glass on a long (~half a km) basin of molten tin where it stretches and cools slowly, pushing the solidifying mass down the basin until the end. Re-Melting it is as far as i know not possible, as the melting is done before the pouring. You can melt the tin again, but not the glass.
I'm guessing that liquid glass gets into some spots/crevices that it cannot be remelted from using built-in heat sources.
It works also with cold/frozen stuff - you can even store snow over the summer if needed: https://www.slf.ch/en/snow/snow-sports/schnee-und-ressourcen...
Interesting! Does this mean that you could use intermittent energy sources, like solar power, to heat up the float glass oven during the day and it woul keep enough heat during the night so that it could keep running in the morning?
Float glass is a continuous process, so it runs 24/7 and too much product is recycled into the smelter at the start again (Funnily, glass waste is actually a scarce resource). Getting the process dialed in to emit the desired quality is a long winded thing as far as i know (not involved in the process myself). Also, the heating is done with gas (LNG/CNG/H2) and electric in combination.
Yes, in theory, but your process control would be terrible, thus your product would also be terrible. Unless you invent something very clever.
But that makes them superb heat battery, which is under used at the moment to store energy.
because of the Carnot cycle, no? it just doesn't make much sense to store energy as heat
A lot of energy is spent to heat things anyway.
How did you find that much insulation quickly and what kind was it?
The Insulation is in place all the time because the process is very energy intensive and the glass over the whole basin is supposed to cool evenly. Just the end is not closed off as the glass is ejected in a continuous endless stream[1].
https://www.blindex.com.br/-/media/blindex/site-content/xx-n...
In some cases they don't have any practical ability to.
The Texas winter storm resulted in a lot of scrap at semiconductor lines due to power loss. There are industries that are completely dependent on grid power. You can't generate enough on site to back up 100% of your operation. Think about how much power one EUV source consumes.
The only reason Samsung is building additional factories in Texas is because the local utilities are effectively treating them as a critical load. The new plants are right by the ERCOT operation center and likely have access to the same cranking paths that the grid uses for black starts.
I have no idea how much power an EUV source consumes, but I’m surprised to hear that it’s more than a handful of kW. What’s the right ballpark?
ASML says 1.3 MW for a single system. So off by three orders of magnitude.
https://www.asml.com/en/news/stories/2024/a-sustainability-m...
Thanks for sharing the link! 1.3MW really isn’t that high of a load though. Thats a very straightforward load to be able to support via diesel generators. As an example, a typical large datacenter uses an order of magnitude more power, yet stays online following a grid outage.
This is the power consumed by the light source for one photo tool on one line.
There are thousands of tools. Material handling robots, compressors, HVAC, pumps, an entire data center within the factory, etc.
It's 1743 horsepower.
Yep, you did the simple math. 2MW generators are rather common place, they even come in portable (as in on a trailer) form.
>They require up to 10.2 gigawatt hours per tool annually
First hit off google.
Assuming they run them 24/7 that amounts to a power usage a bit over 1 megawatt, but they could be pulsed for all I know and use quite a bit more
Power guarantees and insurance.
They are also frequently situated close to reliable power sources such as nuclear or hydro, usually fed by more than one generating station.
A friend of mine lived close to a brick plant his power NEVER went out.
Anecdotal example: The, now closed, Aluminium plant here was build only because the government build a nuclear plant, which is still going. There was a coal plant next to it (closed now) as a backup for when nuclear was not available. Funny thing is most people thought the coal plant was the nuclear plant when driving by since it looks more like a plant, the nuclear dome structure is kind of hidden from the road and since natural water is used as cooling no big towers are close by.
I had the same experience living near a hospital. The whole city has gone out a few times, and our grid was either still up, or is one of the first grids back up.
That was a fun 10 hours or so that we were without power in most of Barcelona. Some barrios had ~some~ power but most of us were without.
The one thing that surprised me is how quickly rumors started about power being out in Portugal (true), France (true to an extent), Belgium (false), and the UK (false). Walking back home from work you heard the names of Trump being mentioned as well.
It's a bit scary to personally experience how quickly people can start panicking. On the other hand the bars that had some power had a great afternoon with tons of beer being served before it got too warm :-)
I am in Barcelona too. The combination of no power and patchy internet was the perfect recipe for rumors to spread, which was fun to observe. I went for a walk since I couldn't work and saw tons of people socializing, drinking beers (more than usual), and young people playing without their phones. It'll be a special day to remember, I think.
Maybe make it a yearly event, turn off all power (like Earth Day) just for the socializing aspect.
Shutting off (anti-) social media would be enough.
I honestly thought we should make it a public holiday and shut down the electricity once a year, not so realistic unfortunately.
If we're ready for it and can prepare, it wouldn't be so bad I'd guess. Trouble yesterday was that we had no idea about the scope, I was in the middle of finishing moving houses, but was promptly held back when I arrived with a van to Barcelona and noticed it wasn't just my town that was without power.
I was actually considering going for a ride on the motorcycle today if the outage lasted another day. Beautiful weather, a tank full of gasoline, and no stress about work? Sign me up!
Spain and Portugal had to deny a fake social viral that said power was out across most of Europe and the Russians were cutting undersea cables.
On the upside, more people at all levels are going to be more prepared if it happens again.
Most people I spoke to around Maresme area yesterday believed someone incompetent made a mistake somewhere, and that was the reason. Some people jokingly suggested cyber attack/Russian probes, but afaik no one took those seriously.
I'm close to a refinery and seeing black smoke coming out 5 minutes later of the blackout was a bit of a scare tbh.
A security guard at my workplace actually told me they had "received a call" about power being out in Romania as well. I'm not sure how that one rumour spread.
Most information was spread by car radios. Rest were people asking and talking to each other.
As long as the beer is cool, you only need CO2! Great off grid mechanism. Or N2 in case of Guinness.
I never have any cash on me. That would’ve sucked.
In my experience, shops you frequent will be sympathetic to that problem and will just let you come back to pay later. And I imagine that any shops that sell stuff that needs to be kept cool were especially amenable to that ...
I spent my last 5 euros on a sandwich and a croissant to eat on the walk home. All in all a pretty decent experience.
Your comment immediately brings to mind this classic of croissant eating in ‘interesting’ circumstances: https://youtube.com/watch?v=wp84sRpM1Js
Add the store on Foursquare, "good blackout sandwich, 5 stars"
How warm is "too warm"? I find cold beer, and cold soft drinks as well, quite repulsive. I usually try to order without ice entirely, and if I receive any bottles or cans, I permit them to come to room temp before consumption.
Refrigeration is sort of a scourge to those who enjoy food. We are so used to simply freezing our esophagus and not tasting anything! Food is kept cold because it lasts longer, not because it tastes better.
Sure, refrigeration makes restaurants and grocery stores possible. But it also presents challenges to cooking and it's very resource-intensive -- think about it -- basically everyone everywhere is always running their refrigeration -- that's a huge load on any system.
I was referring to the fact that generally people prefer cold beer, not room temp beer.
Pedantic, but most beers should be drank (according to the manufacturers and experts) at 7-13c depending on type of beer. Exceptions always exist to confirm the rule. Which is above fridge and around "cellar" temperatures.
Unless you drink Schultenbrau, which tastes worse cold than at room temperature (/s, kind of).
TIL Schultenbrau is a real beer. I thought it was gag beer designed for the New Kids show.
The same goes for wine, by the way: cellar temperature is usually optimal.
> generally people prefer cold beer
Do they? But why? Why would they? Are they simply accustomed to tasting it that way? I should say "not tasting it" because chilling a beverage basically kills its flavours, and they never make it to your tongue correctly.
I would say that many people would be surprised and amazed if they tasted such things as Pepsi without ice, and without coming right out of the fridge.
Unfortunately, any lack of refrigeration really messes up supply chains and will eat into anyone's bottom line, so it's not a realistic goal.
This stuff is panel-tested out of the wazoo. There's a reason bitterness, hoppiness, alcohol levels, differ market by market, even on brands of the same brand/line extension.
Beer also does not require refrigeration throughout the entire supply chain, they happily sit in a warm warehouse waiting to be picked up by an unrefrigerated truck, only to be cooled at the outlet or consumer.
I've always theorized that, in prehistory, the only source of cold water would be recent snowmelt; streams that came fairly directly from a probably-clean source. Warm water could have all sorts of things growing in it. So our ancestors who found cool water to be particularly refreshing and sought it out might have suffered less from waterborne illness.
I can only imagine the difficulties facing technicians and engineers black starting a nation-scale grid.
I operate a microgrid facility in Hispaniola and have wonderfully cooperative users backed by a separately powered communications and control system. Even for us, a facility serving a small neighborhood and farm, a black start must be performed as a careful choreography of systems and loads…and we can just pick up a radio and tell people to turn the main breaker of their house on or off, and to leave their AC units off until we finish bringing everything up. In 12 years we’ve only had to do it twice, but even for us it’s a tedious process.
I am no expert, however with solar power and more grid batteries (I don’t think Spain has many grid batteries) wouldn’t it be possible to charge the batteries and use that to drive a ‘black start’.
Sounds simple, and I appreciate from experience a huge amount of prep and validation needs to be done in the background.
At an extremely abstract level, it is conceptually simple, but most inverters are designed with safeguards to prevent exactly this. They only produce power when the grid is up, and in phase with it.
Because if not sequenced and interlocked, islanding behavior can result in back-powering parts of the grid that are supposed to be disconnected and down for maintenance, or creation of desynchronized islands that then cause more damage when reconnected.
This is absolutely a problem that operators need to solve. Radio synchronization is possible, neighborhood islanding could be beneficial, etc. There are entire village-wide micro-grids in remote places where everyone's inverters do perform these functions, it just hasn't been embraced in monopoly environments yet.
For instance, relays that close isolated parts back onto the main network would need to be phase-aware, to only reconnect if it's safe to do so. You could mandate that all island-mode inverters center themselves around 50.1 or 60.1Hz in the absence of outside influence so there's certain to be phasing opportunities. Distribution networks would need more switches to provide positive isolation during maintenance. And all of these things would need huge amounts of interoperability testing. It's possible, but it has a cost.
Thanks for the excellent insight.
I’m guessing this could be more of a thing in the future to make the grid more robust.
Didn’t Australia have a similar issue a few years back, which Tesla built a large grid battery to resolve, not just for storage but for stability.
This article was published when over half of power was already restored.
99% up now. It's pretty impressive how fast they can get the system up and running again considering how seldom they need to do these types of things.
Initial estimates went from "No idea" to "Could take days".
In the end it took about 12 hours to restore most of the power in two countries. There are some knock-on effects from trains being in the wrong places, but a solid result overall.
This is not correct. The REE press conference held about two hours after the start of the blackout gave an estimate of 6 to 10 hours, which was about right. Any other estimations were misinformation.
Some outlets said days to weeks, but who knows where they got that info.
What caused it? Nowadays when anything like this happens in the EU, I immediately wonder if it was Russian saboteurs. They have certainly been destroying infrastructure here, eg. undersea internet and power cables.
I was wondering if a solar flare could cause something like this. But if it had been that on this occasion, the astronomers would've been warning about it for at least 8 minutes. :-)
It was American saboteurs. First thing that comes to my mind /s
[dead]
[dead]