/*
* arm_irq_memory_barrier()
*
* Ensure all writes to device memory have reached devices before proceeding.
*
* This is intended to be called from the post-filter and post-thread routines
* of an interrupt controller implementation. A peripheral device driver should
* use bus_space_barrier() if it needs to ensure a write has reached the
* hardware for some reason other than clearing interrupt conditions.
*
* The need for this function arises from the ARM weak memory ordering model.
* Writes to locations mapped with the Device attribute bypass any caches, but
* are buffered. Multiple writes to the same device will be observed by that
* device in the order issued by the cpu. Writes to different devices may
* appear at those devices in a different order than issued by the cpu. That
* is, if the cpu writes to device A then device B, the write to device B could
* complete before the write to device A.
*
* Consider a typical device interrupt handler which services the interrupt and
* writes to a device status-acknowledge register to clear the interrupt before
* returning. That write is posted to the L2 controller which "immediately"
* places it in a store buffer and automatically drains that buffer. This can
* be less immediate than you'd think... There may be no free slots in the store
* buffers, so an existing buffer has to be drained first to make room. The
* target bus may be busy with other traffic (such as DMA for various devices),
* delaying the drain of the store buffer for some indeterminate time. While
* all this delay is happening, execution proceeds on the CPU, unwinding its way
* out of the interrupt call stack to the point where the interrupt driver code
* is ready to EOI and unmask the interrupt. The interrupt controller may be
* accessed via a faster bus than the hardware whose handler just ran; the write
* to unmask and EOI the interrupt may complete quickly while the device write
* to ack and clear the interrupt source is still lingering in a store buffer
* waiting for access to a slower bus. With the interrupt unmasked at the
* interrupt controller but still active at the device, as soon as interrupts
* are enabled on the core the device re-interrupts immediately: now you've got
* a spurious interrupt on your hands.
*
* The right way to fix this problem is for every device driver to use the
* proper bus_space_barrier() calls in its interrupt handler. For ARM a single
* barrier call at the end of the handler would work. This would have to be
* done to every driver in the system, not just arm-specific drivers.
*
* Another potential fix is to map all device memory as Strongly-Ordered rather
* than Device memory, which takes the store buffers out of the picture. This
* has a pretty big impact on overall system performance, because each strongly
* ordered memory access causes all L2 store buffers to be drained.
*
* A compromise solution is to have the interrupt controller implementation call
* this function to establish a barrier between writes to the interrupt-source
* device and writes to the interrupt controller device.
*
* This takes the interrupt number as an argument, and currently doesn't use it.
* The plan is that maybe some day there is a way to flag certain interrupts as
* "memory barrier safe" and we can avoid this overhead with them.
*/
void
arm_irq_memory_barrier(uintptr_t irq)
{
dsb();
cpu_l2cache_drain_writebuf();
}
static int
arm32_drain_writebuf(struct thread *td, void *args)
{
/* No args. */
#if __ARM_ARCH < 6
cpu_drain_writebuf();
#else
dsb();
cpu_l2cache_drain_writebuf();
#endif
td->td_retval[0] = 0;
return (0);
}
void
generic_bs_barrier(void *t, bus_space_handle_t bsh, bus_size_t offset,
bus_size_t len, int flags)
{
/*
* dsb() will drain the L1 write buffer and establish a memory access
* barrier point on platforms where that has meaning. On a write we
* also need to drain the L2 write buffer, because most on-chip memory
* mapped devices are downstream of the L2 cache. Note that this needs
* to be done even for memory mapped as Device type, because while
* Device memory is not cached, writes to it are still buffered.
*/
dsb();
if (flags & BUS_SPACE_BARRIER_WRITE) {
cpu_l2cache_drain_writebuf();
}
}
