Discussion about this post

User's avatar
richardstevenhack's avatar

"Guardrails are controls that constrain how AI systems operate within development environments. They may include secure coding policies, approved architectural patterns, dependency restrictions and automated validation mechanisms."

That's not going to be good enough.

The core problem is the existing models are trained on random bad engineering practices because that's how most "software engineers" were trained - or rather developed by osmosis.

Three things need to be done:

1) New models must be developed on solid engineering practices. No "bad code" allowed in their training data. These models must also be more deterministic.

2) These models must be trained as coding assistants which can guide human system designers on proper engineering principles during the process of system development, so mistakes are caught in the design phase.

3) All resulting code and systems must be reviewed by humans and models trained to both identify and remediate poor engineering practices and also poor security practices. Much of the latter is dependent on the former; if the system is developed to be provably correct, it's much less likely to have a security issue.

Without this fundamental change in software development, things are just going to get worse, not better.

Been saying this for twenty five, thirty years. In fact, probably since back during the "expert systems" days of AI. The whole point of "expert systems" was to produce correct results and catch mistakes humans make.

Of course, it won't happen because in essence it would remove the profession of "programmer" from existence to be replaced by AI-aided system designers. And system engineering is harder than programming random code.

No posts

Ready for more?