You can have it integrated with grafana, and it can quickly search logs based on errors or alerts and link them to specific commits quite quickly. My previous company did that a year ago.

If you just vibe AI, it will lull you into thinking you’re doing great.

1. Define what successful AI usage looks like and include metrics and measurements — include AI usage in your story/ticketing system.
2. Define parameters for usage. Will it write documentation? Write tests? Write functions? Classes? Whole features? Will it review code? How are you making sure everyone understands the new code (i.e. you don’t want devs committing code they don’t understand or reviewers passing it)
3. How will you reign in costs? I had 3 devs spend over $1500 (each) in thirty days and am in the process of a explaining this to my COO. Purchasing plans instead of using API keys gives you some natural boundaries for reigning in costs.
4. Create standard processes. Don’t vibe. Have a standard for how your AI-facing documentation is structured. Have standards/templates for how prompts are structured. Have standard prompts with limited scope for specific tasks. For example have a prompt just for review that specifies to review from the standpoint of security and best practices.

Since there’s zero information about what kind of company you’re working at, the following is extremely generalized.

• integrating with monitoring systems, analytics DBs, ticket systems, whatever is used by management, allowing them to ask questions in natural language
• process automation using agentic workflows, e.g. pre-analysis of incoming email queue summarizing / sentiment analysis before the customer support sees it
• provide access to models and model APIs for development workflows and integration into git / ci, allowing to use llm in local development and e.g. setting up something like automated code reviews (not a replacement for human review, only as an addition)
• set up coaching, responsible use, hallucinations, etc.

Whatever you do, take security and data security especially into consideration first, not after:

• consider whether your used provider reuses your data for learning
• consider whether it’s relevant where it’s located (GDPR customers?)
• always set spending limits
• consider your local and your customers data protection laws and regulations that apply to your company (especially in health and financing)

If the person who put you in charge of it knew of your indifference, that’s a smart manager, and you should just trust your gut on this.

Not sure on the size of your team, but you could perhaps form a small subset of AI “champions” who can try to help drive new initiatives. Just some ideas:

1. Start out with foundational usage of AI agents, ensure people have the right mindset and ownership of using the ai agents. Ensure there is always a “human in the loop.”

2. Arrange for some training for people (not just Devs) on the right way of using AI, such as not throwing sensitive information into it. Educate people that it’s far from perfect

3. Get standard boundaries in place for your current and future projects, get context files (agent.md) for your projects so those who do use the agents will be doing so from the best position.

4. See if you can arrange with the higher ups to arrange some company time do a Hackathon with the use of AI to help people see what they can do with it and the challenges they’ll face. You can also take some learnings from it.

The way I see it, you’ve been given the position of head of AI because you’re indifferent. It’s your job now to drive engagement (because that’s what your bosses want) but in a controlled manner. If they put someone that was overly enthusiastic about AI they would end up with a huge bill and lots of AI mess to clean up.

I’m in like 3 AI design forums for some reason but whenever I want to start measuring productivity gains I’m ignored so try that for the least amount of effort required.

First, my condolences.

Second, I think something you can do as “head of AI” is push back on the benchmarks your execs are expecting you to measure. They got them straight from whoever your AI vendor is, and if your executive team is even halfway competent they should understand “these metrics are designed by the vendor to make us spend more”, but a lot of exec teams won’t listen. Still, you should institute your own benchmarks around code quality and delivery speed and talk about them with the exec team even if you have to shoehorn them into the discussion.

The next and probably more important thing that comes to mind is managing how your devs use their new tools. They’ll be able to churn out more lines of code than ever before and “complete” some features much more quickly. Your metrics should not incentivize this if your goal is code quality and stability. I don’t really have much in the way of solutions (other than “manage expectations” and “set the narrative”), because that’s about where in my “head of AI” career I got laid off. Once the numbers came in that we only needed 30% of our current staff they crunched the numbers and I was in the 70%.

Good luck. You’re starting from a deficit because your management team probably already has entertained the thought of trimming some expensive devs from the payroll, and that’s a tough thing to argue against.

It depends on your company.

If you are running public-facing code, security is going to become a huge deal in coming months. So use AI to find all the security holes in your stuff before the hackers do.

If you have lots of databases and APIs for internal use, create a MCP server so agents can do anything with your data and APIs.

If your data is even a little sensitive and you don’t want to send it all to the USA (read up on the CLOUD act), look into running models locally and the hardware investments you’ll need to make.

Nah, show them that it’s an expensive waste of time by continuing to let them run wild with useless but impressive tasks

We don’t even know what your company does? Does it do development? Data analysis?

Without knowing that, as head of AI, I’d first take training to understand what AI is and get advice from an external company as to what it could be good for. “AI” is being used as an umbrella term nowadays for neural networks to stochastic machines and they all have different uses. Understanding valid and invalid usecases is important before deciding when and where to use it in the company.

Damn, I’m in the same boat. I didn’t attend one meeting because ut was too late in My timezone and the others chose this moment to vote me to become the lead of the AI acceleration initiative where we have half a year to come up with and implement things on each step from requirements, through architecture and development to testing on all levels.

And yeah we better come up with something useful because I already know they have to reduce the work force because there is no money and the people who will be left can’t do all of it alone without help.

Pay me 25% and I’ll advise you.

Could you explain what feelings you have about AI, and how you see these feelings as opposing. If you are just uncertain about AI (holding no opposing views), I think you would want to research more. Maybe if we knew more about your teams at your work, and what is being developed. You could honestly just spend an hour a week working on standardizing coding agent availability and licensing/subscriptions, and leave it at that. Either they weren’t looking for someone who was a machine learning engineer, or whoever promoted you is clueless to what AI actually means.

You could just quit since you clearly have no desire to succeed. Or just wait until they find out and fire you.

As a bonus you can blame “AI” either way.