Allow disabling skills for memory agents

What issue are you seeing?

Agents spawned to do memory related tasks sometimes use skills. Which seem unintentional and I'd like to disable that.

What steps can reproduce the bug?

1. Enable $planning-with-files skill from OthmanAdi

planning-with-files

1. Wait until memory worker task is done
2. Agent used the skill as is evident by the files left in the directory.

What is the expected behavior?

Agent does not use skill for memory related tasks.

Additional information

_No response_